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EXECUTIVE SUMMARY 


This report presents the results of a study of various instruction 
methodologies currently in use to teach people how to access and search 
computer-stored bibliographic-type databases in an online mode. The study 
was performed to provide DTIC management with information about practices 
and experiences for DTIC use in reaching decisions about its future course 
of action with respect to educating clientele in DTIC’s access system and 
online databases. 

Discussions with experts in the online community, including members 
of the DTIC staff, led to the selection of programs of the following organi¬ 
zations for examination: 

a. Lockheed Information Retrieval Service (LIRS) (2 programs) 

b. University of Denver Graduate School of Librarianship and 

Information Management (GSLIM) 

c. National Library of Medicine (NLM) 

d. University of Dayton Research Institute (UDRI) 

e. University of Pittsburgh On-Line Training Center (Pitt) 

f. House Information Systems, House of Representatives (H.I.S.) 

g. Congressional Research Service, Library of Congress (CRS) 
as well as DTIC for comparison purposes. 

A set of criterion variables was developed by which the instruction 
programs were analyzed and compared. These variables address four major area 

1. Purposive documentation 

goals, objectives, criteria 

2. Education program 

components, duration(s), availability, location(s), 

participant s 

3. Resources 

human, equipment, facilities, instruction materials 

4. Evaluation/Feedback Mechanisms 


instruments, tests, follow-ups, user groups 







Most programs are found to be weak in purposive documentation that 
limits the bases on which their developers can evaluate their accomplishments. 
Five of the programs rely on lectures as the principal mode of knowledge 
transfer. Pitt and H.I.S. programs integrate lecturing with hands-on practice. 
LIRS's ONTAP and CRS programs provide online self-instruction. Four of the 
programs have a multi-course structure, offering one or more advanced courses 
beyond an introductory one. No course other than the academic one (GSLIM) 
is longer than one week in duration. Only LIRS's ONTAP program is substantially 
location independent; all others require students to travel to either a single 
location or one of several locations. Although six of the courses have no 
course fees, the high indirect costs of travel probably prevent some individuals 
needing instruction from obtaining it. 

Approximately 1 to 1^ months appears to be needed by an instructor to 
plan a program and prepare printed instruction materials for each content 
focus, i.e., a command language or a database. Preparation time for two or 
more foci appears to be cumulat: .'e, for example, 2-4 months for a command 
language and a database. Complex databases require more time, for example, 

3 months for a UDRI manual. Time needed to revise a program and its instruc¬ 
tion materials can vary from 3 days to 1 month dependent on the extent of the 
revision. By comparison, instructors can spend 10 to 100 hours to prepare 
one hour of computer-assisted instruction. 

No "best" student/instructor ratio for online instruction is yet known. 
Ratios of the programs studied vary from 3/1 to 20/1, but they are based on 
the number of terminals available for hands-on practice. Only the Fitt, 

ONTAP, and CRS programs provide each student with a terminal; in most other 
programs, two people share a terminal. Irrespective ot the instruction metho¬ 
dology employed, all programs produce printed materials for students. These 
vary from very brief guides to a database and retrieval options (H.I.S.) to 
replicas of entire lecture content (NLM). 








Six of the programs distribute questionnaires to students at the end 
of a program to obtain a course evaluation. No program has a longer term 
evaluation mechanism, for example, a follow-up six months after the course. 

The customer service desks maintained by HRS, DTIC, NLM, and H.I.S. provide 
feedback to instructors, but only for NLM is this direct since NLM instruction 
staff also perform service desk duties. The user group meetings of DTIC and 
NLM and the update sessions of LIRS are other opportunities for feedback. 

No one methodology can be judged better than the others since they 
differ in purpose, command language, databases, and target audiences. Printed 
manuals or wookbooks appropriate to client needs are seen as essential for at 
least post-course on-the-job reference. The lecture mode of knowledge transfer 
is by no means obsolete, but it is recommended that other media be incorporated 
with it when they can more effectively convey information. For example, 
slide/tape and videotape modules exemplify the "one picture is worth a thousand 
words" adage. These visual/sound media could be components of lectures and/or 
offered for self-paced instruction. Hands-on practice is a sine qua non for 
these instruction programs. Practice totaling 4 to 5 hours per student is 
suggested for programs having the objective of providing the student with 
proficiency to do online searches on-the-job for themselves or end-users. 

GRS's computer-based instruction (CBI) program demonstrates that the 
CBI methodology can be used to convey introductory-level knowledge of a 
command language and databases. CBI is not yet hospitable to teaching advanced 
search complexities. It is also expensive. The LIRS ONTAP approach, that 
uses computers in a more limited way, is worthy of further study. It is self- 
instructional and, with proper printed materials, it might be able to effectively 
replace up to 85% of present instruction programs. 
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INTRODUCTION 

The study that is the subject of this report was commissioned by the 
Defense Technical Information Center (DTIC) because of the confluence of at 
least three factors: 

(1) a managerial interest in periodically reviewing DTIC's instruc¬ 
tion program for its online clientele for adequacy and 
appropriateness, 

(2) growth of the pre-October 1980 demand for instruction that was 
beginning to overload the resources DTIC had allocated to the 
instruction activity, 

(3) commencement of dialup online access to DTIC's databases in 
October 1980 that was expected to introduce greater demands 
for instruction from a more heterogeneous clientele. 

DTIC personnel responsible for customer service are members of the by 

now sizeable "online community." This community consists of 

— "database producers" or "suppliers", organizations that produce 

databases in machine (computer)-readable form, 

— "database vendors" or "search services," organizations that 

purchase or otherwise acquire the machine-readable databases, 
store the databases in a computer system that they maintain, 
and make the databases available to user organizations, 

— "user organizations," organizations that gain access to the 

vendor-stored databases through purchase or some other 
arrangement and have a client or customer relationship 
with the vendor, 

-- "communications utilities," organizations that maintain networks 
that link user organizations' terminals to vendors' computers, 

— organizations of many types, including higher education institu¬ 

tions, consultant groups, individual consultants, and library 
consortia, that offer education programs to enable the users 
to effectively use the computer-stored databases, 

-- publishers and authors who perform a documentation and communi¬ 
cation service for the community. 


A considerable amount of functional overlap may exist within a particular 






organization. For example, database producers are increasingly undertaking 
the task of providing users with one or more education/training courses. 

DT1C is an illustration of a database producer that is also a vendor. All 
vendors provide, directly or through a contractor, some level of education to 
actual and prospective users. 

The various education programs currently being offered reflect the 
offeror's judgment about the needs of their user groups and their education 
responsibility to these groups. Sometimes, perhaps frequently in the case of 
database producers, the education program represents a compromise between no 
program and a greater level of effort that the database producer cannot afford. 

The use of computer-stored databases is becoming increasingly complex 
for users primarily because of the increasing numbers of databases and vendor 
systems that are becoming available. Most databases that had been produced 
in printed form are now also in machine-readable form and stored in one or 
more public access systems. To be sure, some of the databases (e.g., those 
of the Chemical Abstracts Service and the BioSciences Information Service) 
are structurally complex and subject knowledge facilitates use in the printed 
or online mode. Aside from inherent file content complexity, the growing 
need of users to make selection determinations from among available databases 
imposes a new level of difficulty. Vendors are perceiving the need to provide 
instruction about databases applicable to particular subject areas. At least 
some database producers appear to be reassessing their educational level of 
effort because, if users lack sufficient knowledge about a database, they 
may ignore it in preference to one or two others they are familiar with despite 
their unsuitability to the information need. Since payments for use (i.e., 
fees, royalties) are becoming a significant source of income for database 
producers, some are beginning to make greater investments in education programs. 

Most database producers and vendors have chosen to develop an in-housc 
course preparation and teaching capability. A few producers arc obtaining 
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pare or all of their education programs through contracts. Staff ceilings 
and Office of Management and Budget Circular A-76 Revised (March 29, 1979) 
are turning many federal agencies including DTIC to investigate the private 
sector option for the performance of particular activities. 

This study has examined eight instruction programs in addition to 
DTIC*s to give DTIC management information to use in reaching a decision on 
its future course of action with respect to educating clients in the utiliza 
tion of DTIC's online system and databases. 


STUDY OBJECTIVES 


I_L 

An interagency agreement was established between DTIC and the Of lice 
of Personnel Management, Workforce Effectiveness and Development Group, for 
performance of the instruction methodologies study by an OPM specialist. 

The author is a member of the online community who has participated in develop¬ 
ing and teaching a quarter-length course in an accredited library school. 

The agreed study objectives are: 

a. To identify, analyze, and evaluate training programs that 
utilize different modes of instruction to determine the mode 
or combination of modes most productive and cost effective 
for DTIC*s future training programs for users of the Defense 
On-Line System (DROLS). 

b. To review implications of the instruction methodologies for 
DTIC consideration in formulating and implementing future 
education programs. 

DTIC identified particular types of education methodologies and train¬ 
ing programs that were of interest. These were: 

-- computer-assisted instruction 

— programmed texts 

— contractor-supplied training 

-- audiovisual modules 

-- self-teaching modules 

-- lecture augmented by printed materials 

-- beginner/advanced/refresher modules 

Within fund and investigative time constraints, it was agreed that up 
to 8 programs that exemplified the foregoing methodologies or combinations 
of them would be reviewed, compared to the extent possible, and assessed tor 
applicability to DROLS. 
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STUDY METHODOLOGY 

The initial phase of the study focused on selection of the programs to 
be reviewed. Preliminary inquiries were made by telephone about programs 
known to DTIC personnel and the investigator. These inquiries soughL informa¬ 
tion about the methodologies employed. In several instances it was found 
that a reputation for use of a methodology other than the lecture mode was 
in error. Telephone inquiries were also made to several major library consortia; 
library schools, and consultants to "brainstorm" for candidate program selec¬ 
tion. Where uncertainty was expressed about the nature of a program, follow-up 
telephone calls were made. 

It became apparent that the lecture mode of teaching the use of online 
databases is sufficiently pervasive that a modification of the study plan had 
to be introduced. The initial expectation was that many methodologies could 
be found in a relatively "pure" form, that is, used by themselves not in 
combination with others. It was decided to accept programs that used either 
one methodology or one predominantly or a variety of methodologies such that 
the methodological novelty resided in the mix being used. Although the 
lecture mode is pervasive, it was instructive to find that lecture alone is 
being phased out as an instructional methodology. 

Accordingly, the following programs were chosen for review: 

- for computer-based instruction (CBI, frequently used interchange¬ 
ably with "computer-assisted instruction" or CAI) 

Congressional Research Service (CRS)*s SCORPIO program 

- for an integration of several media and for a contractor- 
supplied program 

University of Dayton Research Institute (UDRI)'s FTD 

program 

- for a self-instruction program 

Lockheed Information Retrieval Service (LIRS) f s ONTAP 


program 
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- for Learning predominantly through a hands-on practice program 

House Information Systems (H.I.S.)'s multi-database program 

- for a 50/50 lecture/hands-on program 

University of Pittsburgh On-Line Training Center (Pitt)'s 
program 

-for a reasonably typical lecture-based academic program 

University of Denver Graduate School of Librarianship 
and Information Management (GSLIM)'s course 

- for a mostly lecture-based training program 

LIRS's seminar programs 

- for a federal agency analog to DTIC' s DRQLS program 

National Library of Medicine (NLM)'s MEDLINE program 

Additionally, like DTIG, CRS, H.I.S., and NLM are database producers. 
Like DTIC, CRS, LIRS, H.I.S, and NLM have a vendor or vendor-type relationship 
with user organizations. Other methodological characteristics of these programs 
are discussed in the next section headed Study Findings. 

Contact persons for the programs were identified and telephoned. All 
individuals expressed a willingness to be interviewed and to share their 
education program experiences with the investigator. 

In all instances, information about the programs was obtained by inter¬ 
view. Interviews were conducted at the education sites for all organizations 
except LIRS. The investigator had attended two of LIRS's recent programs. 

For LIRS, the contact person was interviewed by telephone. For the interviews, 
an interview schedule was developed, field tested, and modified. The final 
schedule is shown in Appendix A. In most instances, the interview schedule 
was sent to the contact person before the interview was held. In all instances, 
contact persons were called on one or more occasions after the interview for 
additional or verifying information. 

To comparatively analyze the information collected, a set of program 
characteristics or features was developed. These appear in the left column 
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of Table I (page 68 ). From an evaluation methodology viewpoint, these 
characteristics can be used as criterion variables.^" They are employed in 
this study as the basis for a comparative analysis and assessment of education 
programs and methodologies. It should be noted that assessments do not have 
to be value loaded (i.e., good, bad). Rather, assessments can identify 
presence or absence, or degree of presence, without a value judgment. Alter¬ 
natively, the characteristics can be value loaded. For example, an evaluator 
may predetermine that a student/instructor ratio in the range of 3 to 5 is 
excellent, 6 to 9 is very good, 10 to 15 is acceptable, etc. When ratio data 
are collected for particular programs, they can be rated in terms of the 
pre-established scales. 

A departure from the original study methodology was introduced about 
a month after the study commenced. It had been desired to obtain assessments 
of the effectiveness of education programs from student participants. It 
began to become evident that systematic access to student recipients of various 
programs was not readily attainable. Consequently, a substitute approach to 
student information was attempted with a questionnaire. The 3-page question¬ 
naire is presented in Appendix A in photoreduced form. The questionnaire 
was promoted and distributed by the investigator at two sessions of the 
Annual Meeting of the American Society for Information Sciences in October 
1980 and at a meeting of the On-Line Users Group in Washington, D.C. It was 
also disseminated by the University of Dayton and the University of Pittsburgh 
contact persons in their respective locations. The total return was 16 ques¬ 
tionnaires of about 120 distributed, a disappointing 13%. 

Information from the interviews and the questionnaires was aggregated 
and analyzed. During the interviews, contact persons were asked for copies 
of their training materials. Not all training materials were obtained (e.g., 
Chemical Abstracts' ONTAP materials). Available materials were examined as 
part of the analysis to produce this report. 


C> 
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STUDY FINDINGS 

This section is subdivided into two parts. The first part presents 
brief descriptions of the instruction programs selected for examination. The 
second part contains an assessment of the programs oriented to assist a manager 
in reaching a decision about appropriate methodologies for his/her organiza¬ 
tion's clientele and databases. 


FINDINGS, PART I - INSTRUCTION PROGRAM DESCRIPTIONS 


Information for each of nine programs is presented in the following 

format: 

Type of Instruction 

Principal mode of delivery 
Auxiliary modes of delivery 
Organization providing instruction 
Name and address 
Contact person(s) 

Telephone 

Type of organization (i.e., public, private, vendor, 
database producer, etc.) 

Recipients of instruction 

Class size(s) 

Location(s) of instruction 

Frequency of offering/availability 

Brief description of nature of instruction 

Improvements desired/planned 

References 

Additionally, for several programs, a section entitled Background has 
been inserted immediately before Brief description of nature of instruction. 
For these programs, it was believed important to understanding to include 
either the historic development of the program or information about the 
databases taught in the program. 




For the last data entry. References, citations are given for reports 
and/or published papers given to the investigator by the contact person. No 
literature search about the programs was conducted. 

The program descriptions are presented in this section by a Type of 
Instruction arrangement. The order is from the more traditional instruction 
methodologies to the newer methodologies. By organization, the pages that 
follow contain descriptions for the programs of: 

Lockheed Information Retrieval Service (LIRS) 

University of Denver Graduate School of Librarianship 
and Information Management (GSLIM) 

Defense Technical Information Center (DTIC) 

University of Dayton Research Institute (UDRI) 

University of Pittsburgh On-Line Training Center (Pitt) 

House Information Systems (H»I.S.) 

LIRS ONTAP 

Congressional Research Service (CRS) 

The current program of DTIC is included to provide a baseline for DTIC 


management 


Type of Instruction 

Principal mode of delivery: Lecture 
Auxiliary modes of delivery: 

For introductory course: Printed workbook, 

Online demonstrations, 

Online practice 

For subject seminars (advanced courses) and refresher: 

Printed workbook 

For updates: Sample searches handout 
Online demonstration 

Organization Providing Instruction 

Lockheed Information Retrieval Service (LIRS) 

3460 Hillview Avenue 
Palo Alto, CA. 94304 

Instructors 

Sophie K. Hudnut, DIALOG Training Coordinator 
Staff of instructors, number not disclosed 

(415) 858-3810; (800) 227-1927 (except CA); (800) 982-5838 (in CA) 

Type of Organization 

commercial online services vendor 
also contractor supplying instruction 

Recipients of Instruction 

primarily DIALOG users and prospective users; no restrictions 
Class size(s ) 

Introductory course: up to 12 
Subject seminars: up to 25 
Refresher: up to 25 
Updates: up to 50 

Location(s) of Instruction 

Introductory course: field offices, major cities in U.S. 

Refresher, subject seminars: same as above 

Updates: frequently in a city back-to-back with a major professional 
society meeting 

Also customer locations, see Brief Description below 
Duration(s) of Instruction 

Introductory course: H days Advanced chemical: 1 day 

Subject seminars, refresher: \ day Updates: 2 days 









Introductory course: almost weekly at Palo Alto, Chicago, New York, and 

Washington, D.C. field offices 
several times per year in each of most major U.S. cities 

Subject seminars, refresher: 8 per quarter nationwide, thus each offered 

about 3 times per year somewhere in the U.S, 

Advanced chemical: 

Updates: 2 to 3 annually 

Brief De scription of Nature of Instruction 

This instruction program focuses solely on one commercial vendor's 
access system, Lockheed's DIALOG, and on the databases available through the 
system. If it can be said that there is an "official" instruction program for 
DIALOG, this is it since the program is produced and provided by DIALOG staff. 

LIRS's position on instruction has changed since DIALOG first went 
public. It was initially thought that education programs would be developed 
by educational institutions and database producers. The very slow entry of 
these organizations into the education/training field, customer need for 
training, and LIRS receptivity to providing assistance gradually brought LIRS 
to its present position of taking responsibility for training in the use of 
its system and for providing at least an introductory understanding of its 
databases. 

Five categories of instruction are provided: 

(1) an introductory course called "System Seminar" 

(2) a refresher course called "System Refresher" 

(3) advanced courses called "Subject Seminars" 

(4) one more intensive advanced course called "Advanced Chemistry" 

(5) updates that LIRS doesn't classify as courses but are tutorial 

The System Seminar gives reasonably well balanced instruction in 
(a) the access language (DIALOG), (b) search strategy and tactics, (c) data¬ 
base structure, and (d) integration of these elements. Illustrative uses of 
the access language and illustrative search problems demonstrate the content 
of a variety of databases and retrieval possibilities from them. Considerable 
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emphasis is given to strengths and pitfalls of various search strategy 
specifications. In the workbook that students keep, various aspects of 
search outcomes are highlighted and annotated. 

The amount of hands-on practice that students receive in the introduc¬ 
tory course varies with the availability of terminals at the course location. 
About 3 to 4 hours of online time are allocated per course. When it is given 
in Palo Alto with 6 terminals available, each student may receive up to 1 hour 
of practice (individuals working at the different terminals simultaneously). 
When a single terminal is available, each student may receive only 15 minutes 
of practice; some may get less. When the course is given at a customer's 
location (see below), the amount of practice is determined by the amount of 
time the customer pays for. 

LIRS currently offers 14 courses beyond the introductory: 

- Subject Seminars in Biosciences, Business, Chemistry, Excerpta 

Medica, Government Documents, Humanities, Legal Information, 
Nonbibliographic Databases, Patents, Science/Technology, and 
Social Sciences 

- a Seminar in Search Strategy 

- a Refresher Seminar 

- an Advanced Chemistry Seminar. 

An effort is made to present each of the Subject Seminars and the Refresher 
in each part of the U.S. (to minimize travel expenses and inconvenience) once 
annually, although the decision for a particular seminar in a particular 
location is undoubtedly also determined by expressed customer need. The 
Seminars have neither online demonstrations nor online practice on the assump¬ 
tion that students have this experience and primarily need a better knowledge 
of use of various files in a subject field. Each of the workbooks prepared 
for the seminars presents a discussion of advanced features of DIALOG and 
annotated illustrations of subject searches. 
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LIRS will additionally provide any of the training courses at a 
customer's location for a fee of $300 for a half day, $375 for 2 half-days, 

$500 for 1^ days, and $575 for 2 days. LIRS furnishes one instructor per 
course regardless of the location. Each instructor is a specialist in one 
or more subject areas. Lately, two or more courses, the Introductory course 
and a Subject Seminar, are being offered back-to-back at a location to reduce 
expenses. A salutary aspect of this practice is that a Subject Seminar can 
be given with fewer than 10 students since the major expense has been covered 
by the introductory course. 

The updates are viewed by Lockheed as opportunities for experienced 
searchers to discuss problems and strategies among themselves and with LIRS 
personnel and representatives of database producers. For example, the Fall 
1980 Update held in San Francisco before the Online ’80 meeting featured 7 
subject sessions, each focused around 5 search topics. Arrangements were made 
for relevant database producers to prepare and discuss the strategies they 
formulated for the topics. The Update also included sessions about new DIALOG 
databases, the DIALORDER procedure, and LIRS's private files service; a "What's 
New with DIAI/3G?" summary by the DIALOG director; and a database exhibit at 
which producer brochures were displayed. 

Improvements Desired/Planned 

The Training Coordinator views education as an evolving process and 
states that training materials are under continuous review for revision as 
needed. In September 1980, LIRS announced a major revision in its database 
documentation ("Guide to DIALOG - Databases"). LIRS is reissuing this publi¬ 
cation as separate chapters and it is eliminating preliminary versions of 
chapters that were heretofore distributed as separates until a sufficient 
number of chapter texts could be assembled for a looseleaf volume of the 


"Guide." 
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In 1980, several new Subject Seminars were introduced and the pattern 
is likely to continue in 1981. LIRS courses may also be offered in more 
locations in the U.S. An item in LIRS*s November 1980 newsletter, Chronolog, 
encouraged DIALOG users to become host institutions for DIALOG courses. An 
inducement is two free registrations for each seminar hosted or an equivalent 
amount of connect-time credit. The items specified the need for three tele¬ 
phone lines (for 3 terminals) for the Introductory course. 


References 

"DIALOG System Seminar" 

"Chemical Information Seminar", "Business Seminar," etc. for each subject 
seminar 

"Guide to DIALOG Searching" 

"Guide to DIALOG - Databases" - looseleaf set of chapters, each averaging 

about 30 pages and providing a field-by-field description of the database 
as stored in the Lockheed system with several sample searches 
"Chronolog" monthly newsletter 

Bluesheets, two page summaries of the contents of fields of each database 
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Type of Instruction 

Principal mode of delivery: Lecture 
Auxiliary modes of delivery: Printed material, 

Online demonstration. 

Hands-on practice 

Organization Provi ding Instruction 
University of Denver (GSLIH) 

Graduate School of Librarianship & Information Management 
Denver, CO. 80208 

Instructors 

James K. Foyle, Anindya Bose (Rowena W. Swanson to March j.980) 

(303) 753-2557 

Type of Organization 

private, non-profit, academic institution (graduate school) 

Recipients of Instruction 

graduate students in library/information science 

C lass Size 

20 to 40 

Location of Instruction 

classroom and terminals laboratory 

Duration of Instruction 

9 to 11 weeks (4 hrs/wk lecture, 2-5 hrs/wk lab) 

Frequency of Offering/Availabilit y 
quarterly 

Brief D escription of Nature of Instruction 

This description pertains to instruction that was provided in 1979 
and early 1980. Except for the amount of hands-on practice time allocated 
to students, the methodology is typical of academic instruction programs of 
the time. It is believed that university programs are increasingly providing 
the level of hands-on practice available at GSLIM and perhaps more. 

The program presented instruction in two systems, Lockheed's DIALOG 
and SDC's ORBIT. In both instances, all features of the command languages 


were taught. Contents of databases were not taught on the assumption that 
students obtained information about some of them in earlier courses. Data¬ 
base design and loading into the respective computer systems were explained 
in relation to their effect on search strategy development and retrieval. 

Most lecture periods included extensive online demonstrations of features 
selected by both the instructor and students. Within limits, demonstrations 
were performed by students as well as the instructor. Printed manuals distri¬ 
buted to students contained illustrations of online sessions that were used 
when computer systems went down. 

Student performance was monitored by written tests, one for each 
system, and by search results obtained for 3 projects. Two projects involved 
2 or 3 person teams who selected their search questions from a collection of 
real inquiries obtained from local area libraries. The third project was an 
individual one chosen by the student and approved by the instructor. Students 
selected 2 or 3 of the databases most appropriate to the search question to 
search. Neither classroom demonstrations nor student practice was restricted 
to one, two, or a few files. Students discussed their search results and 
problems encountered in the projects in class so that various experiences 
could be shared. A total of three 30 to 40 minute online practice periods 
were allotted to each student for project work. 

The laboratory is equipped with 3 keyboard-CRT-printer terminals of 
different manufacturers for breadth of experience. It also contains most of 
the manuals, thesauri, and search aids that have been published for online 
databases, including materials for systems not taught during the course (e.g., 
MEDLARS). Students typically would spend 2 to 4 hours of their own time weekly 
in the laboratory working with the materials for their projects. 

Improvements Desired/Planned 

Most student requests were for additional hands-on practice. 

Faculty desired additional terminals (at least two) of different 
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manufactures to give students broad experience for the job market. Existing 
terminals operated at 300 baud. At least one 1200-baud terminal was wanted. 

To provide reasonably thorough comprehension of all -- not only the 
basic -- elements of the DIALOG and ORBIT command languages, no reduction in 
exposure time to the systems was deemed possible. Nevertheless, faculty 
wanted to provide learning opportunities for the BRS system and the New York 
Times Information Bank; MEDLINE is briefly presented, without hands-on 
practice, in a medical bibliography course. 

No funds were available for additional equipment or online time. Some 
thought was given to providing instruction on other systems in short-course 
formats (1-2 weeks of intensive instruction and practice with a different 
pricing structure). Advanced short-course instruction that might focus on 
subject areas and databases for them was also being considered for future 
development. 



i 
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Type of Instruction 

Principal mode of delivery: Lecture 
Auxiliary modes of delivery: Hands-on practice, 

Printed materials. 

Flip charts 

Organization Providing Instruction 

Defense Technical Information Center (DT1C) 

Defense Logistics Agency 
Cameron Station, Bldg. 5 
Alexandria, VA. 22314 

Instructor 

James de Persis 
(202) 274-7206 

Typ e of Organization 

public, federal agency, database producer 

Recipients o f Instruction 

For dedicated system: 85% federal agencies staff, 

15% DOD contractor personnel 

For dialup system: same as above, 70%, contractor personnel expected, 

mix of clerical, library, sci/tech staff members 

Class Size 

8 to 10 

(50 people in a 2-day dialup session in Los Angeles in December 1980) 

Location!s) of Instruction 
Training room in DTIC 

Training facilities provided by organizations in the field 
Duration!s) of Instruction 

For dedicated system: Introductory course: 5 days (majority of courses) 

8 days, 10 days (rarely) 

Intro, in field: 5 days 
Intermediate course: 3 or 5 days 

None held in field through 1980 
Advanced course: 3 days 

None held ir field through 1980 

For dialup system: Introductory course: 2, 3, or 5 days planned 

(instruction commenced in late 1980) 
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Frequency of Offering/Availability 

For dedicated system: Introductory: at least one monthly at DTIC 

In field when number of requests warrants expense 
Intermediate, Advanced: on an as requested basis, 
as number of requests reaches 6 to 10 
For dialup system: Pattern not yet established 

Background 

DTIC produces five databases: 

a. Technical Reports (TR) file 

Contains bibliographic data; abstract, subject terms, and copy prices 
(DD Form 1473 information) for scientific and technical reports pro¬ 
duced by DOD agencies and their contractors. The online file contains 
surrogate information for documents acquired by DTIC during the past 
10 years. (Information for documents acquired earlier is not main¬ 
tained online.) 

b. Current Technical Reports (CF) file 

Contains DD1473 information for documents to be announced in the next 
issue of the DTIC publication. Technical Abstracts Bulletin (TAB). 

After publication of the TAB issue, the data are transferred to the 
TR file. 

c. Work Unit Information (WU) file 

Contains management and technical status data (DD Form 1498 information) 
for activities being performed in DOD Research and Development pro¬ 
grams. A "work unit" is the smallest segment into which R&D efforts 
are divided; for example, a DOD project may consist of a set of tasks 
each of which is composed of several work units. The WU file is a 
"full text" file in the sense that it contains all of the information 
reported on DD1498; this includes agencies and individuals doing the 
work, contract/grant and funding data, a description of work objectives 
and content, subject terms, identification numbers (program element, 
etc.), and security limitations. 

d. Program Planning (PP) file 

Contains management, fiscal and technical status data (DD Form 1634 
information) for DOD R&D programs at the project and task levels. 

An R&D "project" consists of one of several related tasks directed 
to the development of equipment, systems, components, techniques, or 
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to the exploration ol a new lield of knowledge. The PP is a "lull 
text" file in the sense that it contains all of the information 
reported on DD1634; this includes agencies and individuals engaged 
in the project, fiscal data and fund resource projections, a des¬ 
cription of work objectives, plans, and progress, identification 
numbers, and security information. 

e. Independent Research (IRi file 

Contains planning and technical information for non-DOD sponsored 
projects that private organizations voluntarily report to DOD on 
DDC Form 271. The projects are within DOD R&D areas of interest. 

They are components of corporate research programs directed toward 
the development of future technologies, systems, and hardware. 

The data are proprietary and exempt from disclosure under the 
Freedom of Information Act, subsection (b), 5 USC 552. The file 
is accessible only to DOD and other federal agencies that have 
been certified to use it. Certification is further limited to 
need-to-know fields of interest and appropriate security classifica¬ 
tions. 

Instruction on the IR file is only included in the 10-day Intro¬ 
ductory course. Participation is limited to individuals and 
agencies that have received the necessary access certifications. 

Brief Description of Nature of Instruction 

Until the Fall of 1980, the number of organizations that used DTIC*s 
online databases was limited to approximately 100 because they had to obtain 
direct line access (dedicated terminals). Only UNIVAC 100 or 200 terminals 
were appropriate. In October 1980, dialup access became available for which 
there are no terminal restrictions. This expansion of databases availability 
is expected to significantly increase the demand for all levels of instruc¬ 
tion. 

For direct line clients, the training methodology in DTIC*s Cameron 
Station facility consists of lecture extensively alternated with hands-on 
practice. Three keyboard-CRT-printer terminals are in the training room. 

The instruction pattern is to subdivide each morning and afternoon into lecture 
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and online segments. Both lecture and student preparation for practice 
require frequent reference to DTIC's printed search aids. The instructor 
provides sets of 8 to 12 questions that are to be searched for each practice 
session. Online practice is performed by 2 or 3 person student teams; the 
individuals take turns at the keyboard. Students use the approximately 20 
feet of chalkboard in the room to record results of some of the searches that 
are then discussed at the beginning of the next lecture period. 

The 5-day Introductory course is the most often given. It covers the 
command language and the TR and WU databases. The 8-day course adds instruc¬ 
tion for the PP database. The 10-day course includes the IR database; it is 
infrequently held because of the limited clientele for this file. 

The Intermediate and Advanced courses have only been presented at 
DTIC's facility. The Intermediate course has been held for students of the 
5-day Introductory course. The 3-day Intermediate course covers a review of 
the TR and WU databases and training in the PP database. The 5-day Inter¬ 
mediate course includes the IR database. The Intermediate course is thus 
primarily a time extension of the Introductory course. 

The 3-day Advanced course is held mainly to provide instruction on 
changes and enhancements to the system. For example, the Advanced course 
most recently held covered text searching that was introduced into the system 
in 1979. Additionally, the course is a forum for user questions and problems. 

Only 5-day Introductory courses have been held for direct-line clients 
in the field. The coverage is essentially the same as for the course at DTIC 
except that only one terminal is typically available for hands-on practice 
rather than DTIC's three. 

Instruction for dialup clients held through December 1980 introduced 
modifications. The duration for the Introductory course was shortened to 
2 or 3 days and most of the hands-on practice was eliminated. In the only 
experimental 3-day course held, only the command language and a limited 
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explanation oE the TR, WU, and PP databases were covered with no hands-on 
practice. At a 2-day dialup course in Los Angeles, 50 people arrived tor 
training; only 2 terminals were available and no screen monitors tor audience 
viewing of demonstrations. 

All students are given certificates regardless of whether they attended 
a 10-day or a 2-day course. Students complete a DDC Form 345, a course evalu¬ 
ation form, that is anonymous. About 50% of the forms are estimated to contain 
constructive information. The current teaching format of half lecture, half 
practice each morning and afternoon emanates from student requests. The 
organization of the October 1980 "Self-Training Manual" also incorporates 
student recommendations that instructional text be arranged by database rather 
than by system commands. Parenthetically, this text is designed to assist 
individuals who have taken DTIC's courses to train other personnel in their 
organization who will not be taking DTIC's courses. 

Improveme nts Desired/Planned 

DTIC is planning a major revision in its training programs. It commissioned 
a study of trairing methodologies to obtain data for a decision on its future 
training program strategy. 

The dialup service is likely to appreciably increase client demand for 
training. Since the DTIC system can separate access to the classified portions 
of its files from the unclassified, and since the WU and PP databases are not 
available from any other online service, popularity of these files can be 
expected to grow. (The unclassified portion of the TR file is contained in 
the database of the National Technical Information Service that is publicly 
available.) It is also anticipated that the dialup clientele will differ to 
some extent from that for direct-line service. Dialup users will probably 
consist mainly of private-sector organizations with a greater heterogeneity of 
information needs. 


DTIC conducts five annual user group meetings, four at regional loca- 
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tions (e.g., California, Midwest, Florida, Northeast) and one at DTIC in the 
Fall. Attendees at the October 1980 DTIC meeting strongly recommended that 
one day of advanced training be held as part of each regional meeting program. 
Implementation of this recommendation is planned. Coverage will include dis¬ 
cussions of user experiences and problems as well as information about system 
changes. 
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Type of Instruction 

Principal mode of delivery: Lecture 

Auxiliary modes of delivery: Computer-based instruction, 

Workbook, 

Hands-on practice, 

Other printed materials 

Organization Prov iding Instruction 

National Library of Medicine (NLM) 

Bibliographic Services Division 
MEDLARS Management Section 
8600 Rockville Pike 
Bethesda, Maryland 20014 

Instructors 

Four full-time staff (including Patricia E. Healy and Sheila Proudman 
who were interviewed) and one half-time person of the MMS 
(202) 496-6193 

Type of Organization 

federal agency, database producer 

Re cipients of Instruction 

Users or prospective users of MEDLINE and other NLM online databases 
C lass Size(s) 

Introductory called "Initial Training": 18 to 20 maximum 
Abbreviated Initial Training: 50 
Advanced Training: 18 to 20 maximum 

Update: 20 (primarily Technical Resource Persons from regional offices) 
(Update not considered training by NLM staff) 

Location(s) of Instruction 

Initial and Advanced Training: NLM, NLM*s Regional Medical Libraries 
Abbreviated InitiaL Training: NLM 
Updates: NLM 

Duration(s) of Instruction 
Initial: 5 days 
Abbreviated Initial: 3 days 
Advanced: 5 days 
Update: 1^ days 
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Frequency of Offering/Availability 

Initial: monthly at NLM; quarterly at an RML 

Abbreviated Initial: 4 per year planned for 1981 

Advanced: 6 to 7 times per year at NLM, 6 to 7 times at an RML 

Update: annual 

Brief Description of Nature of In struction 

NLM has been training researchers since MEDLARS (Medical Literature 
Analysis and Retrieval System) was first automated in the mid-1960's as a 
batch system. An initial 8-month training program was progressively shortened 
to 6 and then to 2 months. When MEDLINE (MEDLARS On-Line) was activated in 
1971, the training program was redesigned and reduced to 3 weeks. A change 
to the present format was instituted in March 1977. The present format 
consists of: 

a. Self-instruction with MEDLEARN, a CBI program 

b. Initial Training, 1 week lecture + workbook + hands-on practice 

c. Self-administered test after 6 or more months 

d. Advanced Training, 1 week lecture + workbook + hands-on practice 

e. Updates 

Additionally, NLM presented one Abbreviated Initial Training session of 3 days 
in 1980 that did not include hands-on practice. Because it was well received, 

4 Abbreviated Initial sessions are scheduled for ]98], 

MEDLEARN was originally developed by the George Washington University 
Medical Center under contract to NLM. This computer-based instruction (CBI) 
program is currently being modified and maintained in-house. Trainees 
receive an instruction kit and a special password for free use of MEDLEARN 
6 weeks before an Initial Training course begins. They are expected to gain 
initial familiarity with logging on and off and with rudimentary aspects of 
the system before they arrive. Staff of the MEDLARS Management Section provide 
such augmenting assistance as may be needed by telephone. Most (but not all) 
of the fright of terminals is overcome before the trainees arrive for the 


Initial Training course 
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Organizations that wish to use MEDLINE must send at least one person 
to NLM for training. The organization does not get an account number until 
one person has received training. However, because of class size limitations, 
only one person from an organization may attend a course at any one time. 

Both the Initial and the Advanced Training courses are held in a class¬ 
room in the new Lister Hill National Center Building. The classroom is equipped 
with 10 TI Silent 700 terminals; there are no CRTs or TV monitors. The 
instruction methodology consists of a combination of lecture, referral to 
workbook and other handouts, and use of the terminal. Two students are assigned 
to a terminal. The workbooks for each of the courses is detailed and voluminous 
(approximately 300 pages each). They have been designed to encompass the 
classroom content so that students can devote their attention to the instruction 
rather than feel compelled to take copious notes. 

Trainees are expected to gain at least 6 months of work experience before 
applying for the Advanced course. Whey they apply, they receive a pretest. 

The current pretest consists of 10 pages of detailed questions about use of 
the command language and various databases plus exercises to be performed 
online using MEDLINE, CANCERLIT/CANCERPROJ, and CHEMLINE. The pretest responses 
are brought to the Advanced course and discussed on the first day. The pretest 
serves as a screen to preclude attendees without the requisite knowledge. 

Many different types of organizations, not only medical schools, hos¬ 
pitals, and research institutions, have become users of MEDLINE. Over 1200 
unique institutions are now users and about 2000 passwords have been issued. 
Because of the heavy demand for training, NLM tried an Abbreviated Initial 
course for 50 people in 1980. Of 3 days duration, it is held in an auditorium- 
type room and does not include hands-on practice. The acceptance of the course 
by attendees has led NLM to schedule 4 for 1981. 


<• 
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NLM issues certificates to trainees, and trainees receive Continuing 

Education (CE) credits from the Medical Library Assocation. The credit 

allocations are: 

Initial Training: 3.5 CE 
Advanced Training: 3.5 CE 

Auditor of either of above (no hands-on practice): 2.0 CE 

Abbreviated Initial: 2.0 CE 

Abbreviated Advanced: 2.0 CE (see below) 

Credit is given only for courses taught by NLM staff or staff authorized by 
NLM. At present, only one group has been authorized by NLM to provide train¬ 
ing, namely the Regional Medical Library at the University of California, Los 
Angeles. The RML/UCLA teaching staff received its training at NLM and taught 
at NLM under the observation of NLM staff prior to gaining authorization. 

NLM schedules the courses to be taught at RML/UCLA, selects the students, 
provides the instruction materials, forwards the certificates, and certifies 
the students to the MLA after training. 

Only one update session is conducted by NLM staff. This is a lk day 
briefing primarily for Technical Resource Persons (TRPs) from the eleven 
regions. TRPs are selected by the RMLs and may be either members of RML 
staffs or from an educational institution in the region. Updates introduce 
TRPs to new files, file designs, changes in the MESH vocabulary, and anticipated 
system modifications. NLM produces printed update material and gives 150-200 
copies to each TRP. Each TRP is expected to conduct at least one 1-day 
update in his/her region annually at which NLM's material is distributed. 

TRPs and the RMLs are free to conduct additional training courses of their 
choosing. 

NLM courses are limited to NLM system users. Some NLM databases are 
available from Bibliographic Retrieval Services, Inc., and Lockheed Informa¬ 
tion Systems announced the forthcoming availability of some databases. 

Although in actuality many BRS users are also NLM users, NLM has not serviced 
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users solely with the BRS system and does not plan to service solely Lockheed 
users. 

Improveme nts Desired/Planned 

Success of the Abbreviated Initial course, described above, has led to 
a decision to offer this course quarterly in 1981. Additionally, it is planned 
to offer an Abbreviated Advanced course at least once in 1981 to test audience 
acceptability. However, presentation of the Abbreviated Initial course 
indicated the desirability of including online demonstrations observable on 
TV monitors during the lecture sessions. It is planned to incorporate this 
teaching methodology in all Abbreviated courses offered in 1981. 

NLM suspects that, with the diversity of user organizations, a single 
Advanced course may be less needed than a series of advanced one-day workshops, 
each centered on a particular subject area. An Advanced course format that 
groups content in such a way as to permit attendees to select topics and days 
most appropriate to them may be developed and offered in 1981. 

Some thought has been given to grouping trainees to create more homo¬ 
geneous groups and thus perhaps improve learning for all. Some trainees 
exhibit "terminal phobia;" separate classes for them might permit a faster 
teaching/learning rate in other classes. Trainees from different user organi¬ 
zations, for example, hospital libraries, research laboratories, chemical 
companies, have different needs and expectations. It is thought that grouping 
by type of user organization might also improve teaching/learning effectiveness. 

A redistribution of workload in the MEDLARS Management Section may 
occur in 1981. This may result in assigning 2 staff members full-time teaching 
responsibility and giving part-time responsibility to 1^ others. At present, 
each person in the MMS is responsible for correspondence from a region and 
each takes turns at the user service desk. 

NLM staff are not expecting to give courses in the field in 1981. They 
have a very limited travel budget, and they are not permitted to accept payment 
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from user organizations to defray travel costs. Heretofore, field courses 
were given by two staff members. It is felt that this causes too much of a 
mental and physical drain on the instructors. For courses given at NLM, 
system and subject specialists are brought in to teach their specialties, 
thus relieving the regular instructors from continuous teaching for intervals 
of time. 

NLM may increase the number of organizations authorized to teach NLM 
courses. The RML at the University of Nebraska is presently under considera¬ 
tion, and perhaps one or two additional RMLs will receive authorization. 

In a periodical paper, NLM staff announced a future availability of a 
CBI program called TOXLEARN. Work on this program was discontinued because 
of staff shortages. It is not certain whether work on TOXLEARN will be resumed 
in 1981. 

A CBI program called CUEMLEARN has been written and is now being 
evaluated. No date for release to users has been set, 

MEDLEARN at present is only available to users at particular times and 
this may also be the pattern for CHEMLEARN. CHEMLEARN probably requires more 
computer capacity than MEDLEARN. It is the capacity requirements of these 
CBI programs that limit their availability. However, in 1980, NLM replaced 
its IBM 370/158 with a 370/168 and at the close of 1980, new storage units 
were installed. This may facilitate the availability of the CBI programs. 
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Type of Instruction 

Principal mode of delivery: Lecture 
Auxiliary modes of delivery: Slides, 

Workbooks, 

Hends-on practice, 

Supplementary printed materials 

O rganization Providing Instruction 

University of Dayton Research Institute (UDRI) 

Kettering Engineering & Research Labs. 

Room 445 

Dayton, Ohio 45469 
Instructors 

J. Kirk Brashear, Project Director 
Linda M. Swartzel 
(513) 229-3711 

Type of Organization 

public, academic, non-profit institute, contractor-supplied instruction 
Recipients of Instruction 

Individuals designated by the contractor (Foreign Technology Division, 
AFSC, Wright-Patterson AFB); backgrounds of recipients have varied from 
the clerical and paraprofessional to librarian to end-user, a wide range 
usually present in each class; the typical student is a non-information 
professional in the GS-4 to -7 range. 

Glass Size(s) 

Introductory course: 8 to 10 
Advanced course: 6 to 8 
SDI Profiles course: 8 to 12 

Location of Instruction 

two adjacent rooms at the UDRI 

Duration(s) of Instruction 

Introductory: 3 days (Tuesday-Thursday) 

Advanced: \\ days 
SDI Profiles: Ik days 
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Frequency of Offering/Availability 
Introductory: monthly 
Advanced: semimonthly 
SDI Profiles: semimonthly 

The Advanced and SDI Profiles are offered back-to-back, the Profiles 
course immediately following the Advanced course. 

Background 

The database (actually a family of databases) for which this instruction 
program provides access training is called CIRC II (Central Information Ref¬ 
erence and Control II). It is the information system of the DOD Scientific 
and Technical Intelligence Information Support Program (STIISP). The STIISP 
meets the intelligence information needs of DOD's S&T intelligence production 
agencies, namely the Naval Intelligence Support Center, the Foreign Science 
and Technology Center, the Medical Intelligence Information Agency, the Missile 
Intelligence Agency, and the Foreign Technology Division of AFSC where CIRC II 
is housed. 

Each of the five intelligence production agencies identifies documents 

for CIRC II and provides the following information for entry into the database: 

Document identification and attributes 
Geographic content 
Temporal content 

Personalities, facilities, nomenclature, and interrelationships 
Subject content 

The database is subdivided into an unclassified portion, currently avail¬ 
able online during the morning (0830 to 1315 EDT, M-F) and a classified (or 
secured) portion, currently available online in the afternoon (1400 to 1700 EDT, 
M-F). The database is subdivided into several subsets. Included among those 
available mornings are the following: 

PFLN Files (Personality, Facility/Organization, Location, Nomenclature) 
Personality - a file of surnames 

Facility/Organization - an index to organizations, facilities, 
military bases and units, and post box numbers 
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Location - an index to geographic coordinates, cities and 
higher geopolitics , geophysical, and geographic areas 
up to but not including countries, oceans and seas 
Nomenclature - an index for materials, system and hardware 
designations and code names, ship names and classes, 
space and missile event numbers, aircraft registra¬ 
tion numbers, etc. 

UPDT - CIRC II records for documents processed during the current 
year and the previous year 
XLAX - index to S&T translations 

CREF - cross reference list of CIRC II sequence numbers, original 
document numbers, and microfiche film numbers 
PRQL - a library of existing SDI profiles 

The UDRI initially developed and provided the instruction programs under 
a sole-source procurement. Since 1979, it has conducted 18 Introductory 
courses, 4 Advanced courses, and 8 Profiles courses. In the Fall of 1980, 
it received a contract renewal under an FTD RFP. 

Brief Description of Nature of Instruction 

On two counts -- the complexity of the databases to be learned and 
the heterogeneous mix of students -- this instruction situation differs 
appreciably from most others. To promote learning, the instructors have devised 
a methodology that combines the extensive use of slides with lectures and 
extensive terminal practice. 

As indicated in the Background section above, document records contain 
not only bibliographic information but also a variety of other descriptors 
used by the intelligence community coupled with several sets of numeric codes. 

In the Introductory course, students must learn not only an access language 
(a modified version of IBM's STAIRS) but also the structures of several data¬ 
bases. A measure of volume as well as complexity may be suggested by noting 
that the PFLN codes for a single symposium or handbook type of document could 


require 500 lines of printout 
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The instructors have no control over selection of the individuals who 
come for training. This selection is made by the user organizations, 63 in 
number, that consist mainly of federal intelligence agencies and a few of their 
contractors (e.g., Battelle Memorial Institute). For some difficult-to-under- 
stand reason, the majority of persons sent by the agencies are non-technical 
and non-professional. This may have some bearing on the turnover that apparently 
occurs among these individuals, because UDRI has received a constant flow of 
about 80 people per year for the Introductory course and about 100 people for 
the Profiles and Advanced courses. Approximately 200 people are active end- 
users in the 63 agencies at the present time. The instructors see the need 
for course materials and course content that can promote learning despite wide 
differences in background and experience among the students. 

An additional complexity is the existence of separate classified and 
unclassified files. As specified above, files containing only unclassified 
terms are accessible in the mornings and classified files (having confidential, 
secret, and top secret classifications) are accessible in the afternoons. 

The initial thrust in the Introductory course is getting students 
accustomed to working at terminals. On the first morning after administrative 
matters are finished, students are seated at terminals and "stepped through" 
logging onto the system and some simple retrieval. UDRI's teaching facility 
consists of a classroom with audiovisual equipment and an adjacent terminal 
room. The terminal room contains 4 Teletype model 40/8's (keyboard, CRT and 
printer; users more typically have model 40/2*s) and could be cramped with 
10 to 12 people in it. Because of the limited number of terminals, individuals 
in 2 or 3 person groups take turns doing and watching. 

The general pattern of instruction is to alternate 30-minute sessions 
of classroom and online practice. At the. first terminal session, students are 
told the keys to press. Thereafter, thcir'practice sessions pragmatically 
test what they learned in the classroom session. 
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For security reasons, FTD monitors the online usage of the databases. 
Records of users' search strategies and online printouts have been given to 
UDRI for analysis. Some general patterns of search behavior are discernible, 
but an inability to discuss the strategies with the searchers leaves unanswered 
questions. One common pattern is the use of simple search strategies, i.e., 
the failure to take advantage of the retrieval power of codes and Boolean 
combinations. This may result from the inquiry the searcher has received from 
the client, but it more likely results from the searcher's limited experiential 
background and/or an insufficient knowledge of the databases. Another common 
pattern is an infrequent use of HELP messages. Some feedback indicates that 
searchers prefer to seek help in the workbooks UDRI has prepared than from 
online aids. Ms. Swartzel believes the system's error and help messages may 
be underused because they are in language that is not readily understood. 

A significant feature of the Profiles course is the availability of 
PROL, a database of current SDI profiles. Searchers are encouraged .o use 
this database as an aid in profile development. A separate database, PROM, 
exists for profile construction and maintenance. The function of PROM is 
similar to that of SDC's ORBIT database. The command language (and computer 
program) for profile development was written by BRS (Bibliographic Retrieval 
Services, Inc.), a commercial vendor that also employs a modified version of 
STAIRS. Parenthetically, BRS needed 9 months to debug the program that is 
now being maintained by FTD. The Profiles class assumes a knowledge of CIRC II 
system commands and strategy formulation and is limited to the development 
and maintenance of profiles. 

Im provements Desired/Planned 

A drawback in this program is a relationship between FTD and the con¬ 
tractor that has effectively shut the contractor off from feedback from 
students. FTD has instructed users to call in-house FTD monitors who have 
rarely referred callers to UDRI instructors. The in-house monitors are con- 



versant with administrative and security matters but not with database content 
and search strategy. In discussing UDRI's desire to help users with content 
questions, I was informed by FTD staff that this would be looked into. 

Another mechanism for contact with former students could be a newsletter. 
FTD has been issuing a quarterly document of 30 to 50 pages that consists pri¬ 
marily of administrative memoranda. FTD personnel thought of this document as 
a newsletter. They appeared to look favorably on a suggestion of a newsletter 
similar to Lockheed's Chronolog that UDRI is willing to produce. 

The UDRI instructors have given thought to developing two types of 
form letters. The first is targeted for the student's employer at the time 
the student is designated for the Introductory course. It would provide 
guidance to the supervisor on the support the student needs to be an effective 
searcher. The second letter would be a follow-up to the student a short time 
after he/she completed the course. It would solicit feedback on course 
effectiveness and on the search activities being performed on the job. 

Mr. Brashear has been thinking about the advantages that a static 
database of limited size might have for training. With such a database, the 
University of Pittsburgh On-Line Training Center (see pages 39-41 ) is able 
to know in advance the retrievals possible with search queries and is thus 
able to diagnose students' search strategy errors. It is not known whether 
FTD may be willing to create a training file. 
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T ype of Instruction 

Principal mode of delivery: Hands-on online practice 
Auxiliary modes of delivery: Lecture, 

Printed manuals, 

Supplementary printed handouts 

Organization Providing Instruction 

On-Line Training Center (Pitt) 

University of Pittsburgh 
LIS Building, Room 808 
Pittsburgh, PA. 15260 

Instructors 

Dr. Elizabeth E. Duncan, Coordinator 
Patricia J. Klingensmith 
Nina M. Ross 
(412) 624-5218 

Type of Organization 
public, academic 

Recipients of Instruction 

Any person interested in computerized searching; no restrictions 

Class Size 

Not to exceed 10 

Location of Instruction 

Training center at the University 

Duration of Instruction 

3 days (from 1 P.M. Monday to noon Thursday) 

Frequency of Offaring/Availabilit y 

Approximately one session monthly 

Brief Descrip tion of Nature of Instruction 

This education program emphasizes learning of a particular search system 
on the thesis that "a thorough knowledge of the system is the basis for effi¬ 
cient and intelligent searching." 

At present, only two systems arc taught: Lockheed's DIALOG and SDC's 
ORBIT. Only one system is taught per training course. The course for each 
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system is available in alternate months. For example, in January 1981, 

ORBIT was offered and in February, DIALOG. 

Although the lecture mode is used to begin each course, online work is 
begun as soon as possible. Online work constitutes about 50% of students’ 
activities on the second and third days. For this reason, instruction is 
confined to the On-Line Canter that is equipped with 10 student stations, 
each having a keyboard-and-printer terminal (TI Silent 700), an instructor's 
terminal equipped with a CRT, and three 21-inch TV monitors. The class size 
is limited to 10 to ensure that each student has his/her own work station. 

The course consists of introduction to, and then drill-and-practice 
on, log on and search protocols. Then students are given a battery of 
questions (12 to 15) to search in a particular database. Instructor assist¬ 
ance is available if needed during the period that students develop their 
search strategies and execute them. Search outcomes and pros and cons of 
strategy alternatives are discussed in primarily a lecture mode. This is 
followed by further instruction in, and demonstration of, more complex aspects 
of the search system followed by another battery of questions for students to 
apply to a different database followed by discussion of the new search results. 

Students are given a workbook for the search system that includes a 

[ glossary and detailed explanations and illustrations of elements of the 

search system. They are also given content descriptions of the various data- 

[ 

j bases that are mainly reproductions of materials issued by the commercial 

f 

! vendors. The most difficult level of learning for the students is the pro- 

I 

i cessing of a new set of search questions in databases that were not previously 

discussed in class sessions. This requires them to use their printed materials 
and is a test of their ability to work independently as they would be required 


to do as search librarians 


J 
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The amount of online time that is used in this instruction program 
might be very costly except for a computer technique employed by the program 
designers. DIALOG and ORBIT emulators have been written for a PDF VAX located 
in the University’s computer center. The emulators were written by a computer 
science student who updates them as features of the DIALOG and ORBIT systems 
are added or changed. Tapes containing information on a small number of 
documents, typically 1500 to 3000, are loaded in the Pitt computer instead of 
entire databases. Thus teleprocessing (e.g., TYMNET, TELENET) and computer 
costs of commercial vendors are not incurred. A total cost of $10/hour was 
estimated for use of the VAX system. One hour of student use of the commercial 
system is included to demonstrate effects of complete databases and timesharing 
with other users on computer system responsiveness and retrieval magnitudes. 
Improvements Desired/Planned 

A larger classroom is planned with a capacity for 20 to 24 terminal 
workstations. A larger class size (15) will then be tried. Dr. Duncan does 
not want to lose the personalized instruction aspect of the present program 
that would occur with too large a class. 

Little change is seen as needed in the instruction materials. The one 
program of instruction on the access system (DIALOG, ORBIT) is considered 
sufficient and appropriate for this Center. In-depth education about particular 
databases and about the most effective applications of the access systems to 
particular databases is considered the province of each database producer. 

No advanced course on access systems per se is believed necessary. 

References 
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Type of Instruction 

Principal mode of delivery: Lecture integrated with hands-on practice 
Auxiliary mode of delivery: Manuals 

Organization Providing Instruction 

House Information Systems (H.I.S.) 

Committee on House Administration 

U. S. House of Representatives 

House Annex 2 (2nd & D Streets, S. W.) 

Washington, D. C. 20515 

Instructors 

V, Douglas Hines, Head, Education and Training Systems, directs an 
instruction staff of four 

(202) 225-0217 

Recipients of Instruction 

Members of Congressmen's staffs (mainly legislative assistants or LAs) 
and House Committee staffs (about 1000 people per year) 

Class Size 

6 to 8 maximum 

Location(s) of Instruction 

Training room in House Annex 2 

Training room in Longworth Building (House office bldg.) 

Duration(s) of Instruction 

LEGIS, SCORPIO: two 2-hour sessions 
JURIS: one 2-hour session 
Advanced LEGIS: one 2-hour session 
SOPAD, FAPRS, MBIS: one 1% hour session 

Frequency of Offering/Availabilit y 
Dependent on demand 

Up to approximately 12 classes weekly is within planned workload 
Ba ckgrou nd 

House Information Systems (H.I.S.) was created in 1971 and provides 
computer support to the House of Representatives. H.I.S. has grouped its 
services that furnish access to online databases under the name MIN - Member 
Information Network, Currently, 6 databases are available: 


-L - - -- , _ . 
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LEGIS (Legislative Information and Status System) 

Contains status information on legislation from the 93rd 
Congress to the present 

SCORPIO (Subject-Content-Oriented Retriever for Processing 
Information Online) 

System contains 7 databases; see description of CRS's 
SCORPIO training at pages 51-56 

JURIS (Justice Retrieval and Inquiry System) 

Contains opinions of the Supreme Court, Courts of Appeals, 
other Federal Courts, the U.S. Code, other federal regulations 

SOPAD (Summary of Proceedings and Debates) 

Contains information about House floor proceedings, floor 
schedules, and summaries of vote actions 

FAPRS (Federal Assistance Program Retrieval System) 

Is the online version of the Catalog of Federal Domestic 
Assistance, contains federal loan and grant program data 

MBIS (Member Budget Information System) 

Contains budget tables prepared and updated by the House 
Budget Committee including data by program groupings, 
action on appropriation bills, aggregates by spending 
committees, definitions of terms, and early warning reports 

H.I.S. produces the LEGIS, SOPAD and MBIS databases. The Congressional 
Research Service, Library of Congress, produces SCORPIO. The Department of 
Justice produces JURIS. The Office of Management and Budget has the responsi¬ 
bility for maintaining FAPRS. 

Other activities of H.I.S. include development and maintenance of an 
electronic voting vstem (in continuous operation for over two years), assess¬ 
ments of office automation needs and aid with acquisition and installation of 
computer equipment and services, provision and maintenance of text processing 
software, provision and maintenance of a conversational graphics system called 
IMAGE, and operation of a central computer facility for the House. 
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Brief Description of Nature of Instruction 

A distinguishing feature of the H.I.S. instructional approach is the 
emphasis accorded to hands-on practice by the student in contrast with 
instructor-performed online demonstrations or lesser amounts of online practice 
common in other methodologies. Other differences are: (a) the short training 
times considered sufficient and (b) the simple structure of some databases 
(SOPAD, FAPRS, MBIS). 

For two of the databases having complex content structures, SCORPIO and 
JURIS, instruction programs have been developed by their federal agency pro¬ 
ducers. The instruction programs of the Congressional Research Service and 
the Department of Justice are available to the same user groups as those of 

H.I.S. 

The content of the H.I.S. training manuals suggests that individuals 
are being provided with bare essentials to enable them to perform a few 
frequently occurring types of retrieval as the need for information arises. 

Most of the trainees are not expected to be searchers. Many of the trainees 
may access the databases only occasionally. Trainees are apparently forced 
by the pressure of their work to cancel even the short scheduled sessions 
with some frequency, a further indication of their relationship to searching 
vs. other activities. Additionally, users may obtain search assistance at 
any time from the H.I.S. User Assistance Office (UAO) that has a staff of six 
people who work in shifts to ensure that assistance is available at any hour 
the House is in session (including early morning hours). UAO staff go to 
Congressmen's offices to provide assistance when telephone help doesn't suffice. 
Persons trained in the systems may request retraining as they deem it necessary 
to refresh their knowledge. 

All of the four instructors are competent to teach the use of all of 
the databases. Each may be assigned up to 8 hours of teaching per week. 
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Each course is taught by a single instructor. The instructors have found that 
trainees learn faster when two share a terminal. The Hazeltine 1510, A CRT 
and keyboard, is the terminal in current use. The learning stations do not 
have printers by trainee preference. It is thought that they may not wish to 
have hard-copy records of their learning errors. Although many Congressmen's 
and committees' office terminals are equipped with printers, it is believed 
that many of this user group query files only to obtain one or two items of 
information; they don't need a record of the response. Trainees are instructed 
in use of the print option and they may request a printout of their terminal 
use that is done at a printer in another location. 

Despite notable uses of information technologies by the House, many 
of the staff are slow to adopt the automated mode of searching for informa¬ 
tion. H.I.S. has used the education channel 6 of the House's cable TV system 
to advertise the availability of training for LEGIS, its most heavily used 
database. The videotape prepared by’H.I.S. is broadcast twice weekly, but 
the response from the TV promotion suggests that few of the target audience 
are watching (or are interested). 

Improvements Desired/Planned 

At present, the User Assistance Office is reactive, responding to requests 
for help. Thought is being given to monitoring the usage of databases by 
persons given training and to make follow-up inquiries at least of users in 
the lowest quartile to gain an understanding of reasons for low usage. 

H.I.S. is readying a new version of its printed materials for its 
users. Heretofore, printed materials have been distributed as separates. 

In 1981, a looseleaf binder containing all of the manuals and quick-reference 
cards will be made available. 

Individuals in Congressmen's state offices have access to the H.I.S. 


databases although not much remote use of them has been made. A trend, however, 
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is being discerned of greater use by Congressmen of home office siaffs. In 
anticipation of requests for training at locations other than the two class¬ 
rooms, is examining mechanisms for training at a distance. These mechanisms 

may also be useful to people whose schedules do not permit attendance at the 
live training sessions. 

One methodology is the use of audio tapes as replacements for classroom 
instructors. Two tapes, one for LEGIS and one for JURIS, have been produced. 

They are to be evaluated; they have not yet been distributed. The sound portion 
contains, in essence, the same information that an instructor is likely to 
include in a lesson plan. The trainee is expected to be at a terminal. The 
tape instructs the trainee in the use of the terminal and the database manual. 

The trainee is "walked" through elementary features of the command language 
and the database and is continuously given exercises and searches to perform 
at the terminal followed by verbal descriptions of what he/she should have 
done. 

A second methodology is computer-based instruction (CBI). H.I.S. 
staff explored three CBI software systems -- PLATO of Control Data Corp., 
SCHOLAR/TEACH of Boeing Computer Services, and IIS (Interactive Instructional 
System) of IBM. SCHOLAR/TEACH 3, Version 4 is currently being used to write 
a CBI program that teaches the legislative process. This program, when finished, 
will not replace any of the online training. It could, however, be a precursor 
training module for new Congressional staff. Knowledge of the legislative 
process is assumed in training for LEGIS, SCORPIO, and other databases. 

Other uses of CBI are being sought that will complement rather than replace 
current instruction in the use of online databases. 

References 

"The FAPRS How-To Manual," June 1980. "The MBIS How-To Manual," May 1980. 

"The JURIS How-To Manual," [1980]. "The SOPAD How-To Manual," [1980]. 

"The LEGIS How_To Manual," [1980]. 

"The Advanced LEGIS How-To Manual," [1980]. 


47 - 


Type of Instruction 

Principal mode of delivery: Self-instruction online 

(a form of computer-aided instruction) 

Auxiliary modes of delivery: Lecture, 

Supplementary printed materials 

Organization Providing Instruction 

Principally library schools, but need not be limited to academic institutions 
in the near future, for GAS materials, probably chemistry departments 

Developers of Instruction Materials 

Lockheed Information Retrieval Service (LIRS) 

3460 Hillview Avenue 
Palo Alto, CA. 94304 

Chemical Abstracts Service (CAS) 

P. 0. Box 3012 
Columbus, Ohio 43210 

Contact Persons 

For LIRS: Sophie K. Hudnut, DIALOG Training Coordinator 

(415) 858-3810; (800) 227-1927 (except CA); (800) 982-5838 (in CA) 

For CAS: Diane Yoder, Manager, User Education 
(614) 421-6940 x2945 

Type of Organization 

LIRS: commercial online services vendor 
CAS: not-for-profit, database producer 

Recipients of Instruction 

LIRS: individuals with a rudimentary knowledge of DIALOG, although more 
skilled persons may find this a useful refresher 
CAS: individuals needing in-depth knowledge of how to use CA databases 

Class Size 

primarily the individual learner 
Location of Instruction 

any terminal with access to the Lockheed computer system 
Duration of Instruction 


varies, determined by the individual’s learning needs and preferences 
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Frequency of Offering/Availabilit y 
an the learner's option 

Brief Description of Nature o£ Instruction 

This mode of instruction has been named ONTAP by Lockheed, its principal 
developer; this is an acronym for ONline Training And Practice. 

Currently, there are three ONTAP files in the Lockheed system: 

File 201, ONTAP ERIC 
File 204, ONTAP CA SEARCH 
File 231, ONTAP CHEMNAME 

Lockheed has plans to provide ONTAP files for each of the databases in its 
system, but implementation of these plans does not seem to have a top priority. 

ONTAP files are small samples of the respective databases. By working 
with a file of known content and manageable size, instructors can formulate 
exercises and search questions that they can apply to the file, examine the 
response, and modify to fit aspects of search strategy and/or file content they 
want to demonstrate to students. Then they can assign these exercises and 
questions to students and be able to judge from students' responses the level 
of student learning and expertise. Alternatively, questions and answers can 
be provided in a workbook usable in a self-instructional mode. With large 
and/or open-ended files, instructors either couldn't have complete foreknowledge 
of the response or lists of citations would be too lengthy for practical 
inclusion in a workbook. 

Lockheed produces the tutorial printed material for ONTAP ERIC and the 
Chemical Abstracts Service issued a 250-page manual for Files 204 and 231 in 
September 1980. 

ONTAP ERIC is composed of the 1975 records of the ERIC database. It 
contains 32,120 citations that appeared in the 1975 issues of Resources in 
Educ ation and Curre nt Index to Journ als in Education. Lockheed obtained the 
cooperation of 15 professional librarians in user organizations to evaluate 
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29 test questions and search responses. The questions are divided into three 
levels of complexity: simple, medium, and difficult. 

Students are given a description of ONTAP ERIC that includes the list 
of the test questions and instructions for using the database in an online 
self-instructional mode. The description also contains references to other 
printed materials about the database. The self-instructional program is stored 
in Lockheed's computer. The student selects a question, enters a search strategy, 
and obtains a result. He/she can then compare his/her result with the response 
developed by the professional panel. The comparison can be by numeric count 
(Number of citations in common) and by citation record for references not 
retrieved by the student searcher. By diagnostically examining the latter 
records, the student is supposed to be able to identify deficiencies in his/her 
search strategy. All DIALOG system features can be used in ONTAP ERIC except 
SEARCH-SAVE and PRINT (the offline print command; the online print command is 
TYPE). 

The CAS files consist of 15,700 records that were published in Issues 
15 and 16 of Ch emical Abstra cts Volume 87 (two weeks of October 1977). In a 
test of the CAS manual, two chemistry students at Otterbein College and Roches¬ 
ter University, respectively, were taught the logon procedure and then worked 
independently with the manual and CAS search aids for two weeks. The manual 
covers fundamental concepts of online searching, comparisons of manual and 
computerized searching. Boolean operators, the command language, the structure 
and content of the CA databases, indexing policies and practices, and search 
strategy development. The largest portion of the manual gives suggested 
search strategies for about 12 practice problems. The CAS tutorial does not 
include the relevance/recall feature in ONTAP ERIC. 

This is not computer-assisted instruction in the conventional use of 
the term, but the ONTAP concept is certainly classifiable as a form of 
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computer-aided instruction in that computer processing is an integral part oi 
the teaching/learning process. 

LIRS charges $15 per connect-hour for use of the ONTAP files. CAS does 
not levy a royalty fee on its ONTAP files. LIRS gives 2 hours of free practice 
on ONTAP files to those who enroll in its introductory course, the System 
Seminar (see page 13). 
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Type of Instruction 

Principal mode of delivery: Self-instruction online 

Computer-based instruction (CBI) 

Auxiliary modes of delivery: Video tape, 

Printed materials. 

Demonstration searches 

Organization Providing Instruction 

Information Systems Section (CRS) 

Office of Automated Information Services 
Congressional Research Service 
Library of Congress 
Madison Building, Room LM 222 
Washington, D. C. 20540 

Instructor 

Jeffrey Griffith 
(202) 287-6447 

Type of Organization 

public, federal government, database producer 
Recip ients of Instruction 

Congressional staff members, Congressional Research Service analysts 
Class Size 

Pre-instruction session: 3 to 12 people 
Instruction: 1 person per terminal 

Loca tion of Instruction 

Terminal room in ISS office 

Duration of Instruction 
Learner dependent 

Frequency of Offering/Availabilit y 

Learner dependent within system scheduling possibilities 

Background 

This instruction program trains in the use of the SCORPIO system 
(Subject-Content-Oriented Retriever for Processing Information Online). The 


system currently contains the following databases: 
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Congressional Record (CR) Files 

The CRs for the 94th (2nd Session), 95th, and 96th Congresses 
(1976, 1977-1978, 1979-1980 respectively) that contain abstracts 
of CR pages with floor debates indexed by bill numbers, dates, 
names of members of Congress, and subject terms. These are 
searchable elements. The databases are produced by Capitol 
Services, Inc. 

Bill Digest Files 

Data from the publication "Digest of General Public Bills and 
Resolutions" for the 94th, 95th, and 96th Congresses. These 
files permit a bill's passage through Congress to be monitored. 
In addition to an abstract, bill digest, and revised digest, 
the files contain information on bill sponsor and cosponsors, 
official title, short title, committees of origin, referral, 
and reporting, floor actions, companion bills, and public law 
numbers. 

Major Issues File 

Titles, subject descriptors, and numbers for over 300 papers 
called "Issue Briefs" written and kept updated by CRS staff 
members on major public policy issues. Each Brief contains 
a definition of the issue, background and policy analysis in¬ 
formation, references to Congressional actions, and a biblio¬ 
graphy. 

Citation File 

Abstracts and bibliographic data for periodical articles, 
pamphlets, GPO publications, UN documents, lobby-group documents, 
and other publications selected by CRS staff for their relevance 
to major public policy issues. Searchable elements include 
LIV (Legislative Indexing Vocabulary) subject terms. 

LC Computerized Catalog 

Bibliographic data extracted from the MARC database for books 
in the English language received in LC since 1969, books in 
French received since 1973, and books in all Roman alphabets 
received since 1977. 





National Referral Center Resources 

Name, address, telephone number, topics of concern, collection 
and database holdings, publications, and dissemination services 
for organizations willing and qualified to provide information 
on physical science, technology, and social science topics. The 
file is maintained by the National Referral Center. 

General Accounting Office Recurring Reports 

Abstracts and data on agency names, Congressional recipient, 
mandatory legislation, oversight committees, frequency, date, 
and subject content for all recurring reports required of federal 
agencies by law. GAO produces the database. 

GAO Information Systems and Sources 

Abstracts and data on agency name, contact point, Congressional 
responsibility, funding category, and subject content for infor¬ 
mational documents (e.g., annual reports) and information systems 
(e.g., bibliographic systems, management information systems) 
produced or maintained by federal agencies. GAO produces the 
database. 

GAO Program Evaluations 

Abstracts and data on the sponsoring agency, managing agency, 
program name. Congressional recipient, public availability, and 
‘subject content for about 1700 federal programs that produce 
evaluations of their activities. GAO produces the database. 

Brief Description of Nature of Instruction 

Except for a brief non-CBI beginning, this instruction program relies 
entirely on computer-based instruction (with human assistance if requested) 
to train students in the use of the first four databases listed above, 
namely the Congressional Record, Bill Digest, Major Issues, and Citation 
files. The program uses the PLATO system marketed by the Control Data Corp. 

Trainees are usually given a 45-minute briefing in groups of 3 to 12. 
The briefing includes a 14-minute TV film in which Jeffrey Griffith presents 
introductory information about the four databases, shows some of the source 
material about which information is extracted and stored, and mentions the 
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need to learn search strategies to profitably use the databases. Printed 
material about SCORPIO is distributed including a Survey of Interests form. 

This form enables the trainee to know in advance the learning options available 
to him/her and to reach a decision on topics for which instruction is desired. 

The CBI program is menu driven, and the Survey of Interests material 
reappears on the CRT, forcing a selection if the trainee is to proceed. A 
short routine, "Introduction to SCORPIO Commands" that ends with 7 test questions 
that must be answered, precedes learning modules about the databases. This 
ensures a basic knowledge of the command language that is needed for database 
manipulation. 

The CBI program was written by Control Data Corp. staff under contract 
to CRS. A first draft was written between January and March 1980 and was 
tested in June and July. Mr. Griffith produced a program revision in August 
to reduce the run time. (The first draft also rambled, had too little student 
involvement, and was boring in my judgment based on a small portion of it 
that I worked with.) CDC staff produced the present version in September 
and October 1980. 

I examined the program primarily via the modules of the Citation file. 

I used approximately 1\ hours for 7 modules. Satisfying features of the 
program include: 

a. The appearance of check marks beside the options in a menu that 
are selected so that selections don't have to be remembered. 

b. Reinforcements through restatements appearing on the screen after 
test questions are answered correctly to insure that, should the 
answers have been guessed at, correct learning can be enhanced. 

c. Highlighting of data elements of a record by enclosing each, in 
turn, in a box that initially flashes during an explanation of 
the data element. This has the effect of helping the trainee 
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locate the placement of the data element in relation to other data 
elements as well as seeing how it appears in a record (i.e., whether 
abbreviated, with words or numbers, etc.). 

d. Brevity of instructional messages with many requirements for student 
response. This contrasts with the first version that showed long 
paragraphs of text, a characteristic L also observed in several PLATO 
programs for other subject matter. Most of the instruction is now 
packaged in 1 to 3 sentence sequences that seem about right for 
attention and learning. 

e. Frequent insertion of questions either requiring the trainee to use 
the keyboard or touch an area of the screen. The change from one to 
the other provides a useful diversity, 

A frustrating feature was a limitation in the assistance available in 
HELP routines. I tried HELP for an explanation of RETRIEVE. The present pro¬ 
gram gives general messages and requires the trainee to progress through a 
fixed set of messages that supposedly will include desired information. In 
one use of HELP, I never found an explanation for an error I made and even 
though I could proceed through the instruction program, this created a sense 
of dissatisfaction. 

CRS usually schedules 1 or 2 hours per terminal session for a trainee 
rather than longer periods because of the fatigue factor. I experienced a 
fatigue point after 55 minutes and left the terminal after 2\ hours because 
of mental and physical fatigue. 

Several practice problems are given to the trainee after he/she completes 
the modules selected for study. These may be worked online. They give the 
student a more thorough yardstick for measuring the amount of learning that 
has occurred than mere response to the program's questions does. 

The non-CBI beginning is a 14-minute color TV program that features a 
cast of one, Mr. Griffith. The film holds the interest of the viewer because 
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it is not merely a replica of a lecture. Through verbal description and visual 
images, the film introduces the viewer to the SOORPIO system and the content 
scope of the online databases. It gives the viewer a cursory foundation for 
choosing appropriate CBI programs for detailed study. Mr. Griffith and another 
staff person spent a combined total of about one week on the script. The staff 
person worked about half time for an additional 7 to 8 week period on the 
storyboard, assembling materials, and directing and editing the film. The 
film was shot in two days in a GAO studio. 

Improvements Desired/Planned 

Several changes and additions were made to the command language in 1978 
and 1979. An adjacency capability will probably be added in 1981. Several 
additions may be introduced into the CBI program to reflect the 97th Congress 
(1981-1982). No other major system changes are anticipated in the near future. 
Minor modifications are expected to be provided to trainees by means of 
printed materials. 
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FINDINGS, PART II - INSTRUCTION PROGRAM COMPARISONS 

Table I presents a comparison of the instruction programs in four 
major areas: (see pages 68-75) 

1. Purposive Documentation 

goals, objectives, criteria 

2. Education Program 

components, duration(s), availability, location(s), participants 

3. Resources 

human, equipment, facilities, instructional materials 

4. Evaluation/Feedback Mechanisms 

instruments, tests, follow-ups, user groups 

Since little has been published on the evaluative analysis of education 

programs for information professionals, some comment about the elements of 

Table I may be helpful. In a paper written by the investigator several years 

2 

ago, characteristics of evaluation methodologies were discussed. In the 
terminology of that paper, the elements of Table 1 are criterion variables. 

An education program designer could select appropriate variables and impose 
qualitative or quantitative desiderata on them, for example, a student/instructor 
ratio in the range of 10/1 to 15/1. These specifications could then serve as 
program design guidelines as well as variables by which programs could be 
periodically evaluated. The elements of Table I were developed for online 
instruction programs. Different elements may be needed for other types of 
courses. 

1. PURPOSIVE DOCUMENTATION 

Most programs are weak in purposive documentation. Typically, only 
very general overall goals are stated. Only GLSIM and NLM have produced at 
least partially measurable objectives. GSLIM course syllabi are required to 
include specific objectives and exit skills students are expected to acquire. 

This documentation is a typical requirement in academic institutions. NLM's 












documentation is more extensive; objectives have been developed for each lec¬ 
ture. Most of the objectives can be performance tested. NLM's "System Mechanics" 
lecture, for example, has 5 objectives, one of which is "Describe the purpose, 
entry format, and results of the use of the following ELHILL commands: TIME, 

CAPS, ...etc. " (28 commands are listed). The education community has been 
promoting the use of measurable objectives as an aid to program design and 
evaluation. It can be contended that courses without prespecified directions 
may be less effective in conveying need-to-know information. It is certainly 
true that the absence of objectives precludes a measure of accomplishment 
(unless this is created after the fact). 

2. EDUCATION PROGRAM 

Four of the programs have a multi-course structure; the remainder are 

either single course programs (GSLIM, Pitt) or they have a single course per 

database (H.I.S., ONTAP, CRS). Each of the four has, minimally, an introductory 

course and one advanced course. LIRS is the biggest course producer with 12 

subject seminars in such fields as Biosciences, Business, Government Documents, 

and Nonbibliographic Databases, an advanced search strategy seminar, a system 

refresher seminar, and periodic system updates in addition to its introductory 

System Seminar. It is likely that LIRS will introduce 2 or 3 more subject 

seminars in 1981. NLM has experimented more with course content, duration, 

3 

and format than any of the other organizations. 

In duration, all short courses differ markedly from for-credit academic 
courses. None exceeds a one-week period. NLM, that had an 8-month training 
program when MEDLARS (Medical Literature Analysis and Retrieval System) was 
first automated in the mid-1960 1 s redesigned to a 3-week program for MEDLINE* s 
debut in 1971 and changed to its present multi-course format in 1977. NLM is 
now exploring modifications of its 5-day advanced course to shorter, specialized 
segments so that people will be able to attend only those portions of interest 
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to them. To some extent, a 5-day course of 35 to 40 hours provides 60-70% of 
the instructional contact time of an academic course. Whether concentration 
on a topic for a shorter period or internalization of the topic over a longer 
period produces greater learning and retention is a debatable issue. Several 
short-course instructors (DTIC, UDRI) expressed concern about inadequacy of the 
exposure for adequate learning. On the other hand, periods of several hours 
appear sufficient for H.I.S. trainees and possibly for CRS trainees. Complexity 
of the databases and the command language, and perhaps most importantly the 
learning purpose of the trainee, can materially affect the amount of exposure 
perceived to be sufficient. 

At present, only LIRS's ONTAP programs are substantially location 
independent (dependent only on the availability of a terminal). Of the instruc¬ 
tor-based courses, LIRS's evidence the greatest mobility. In addition to 
offering courses on a regular schedule at field offices and other facilities 
t' roughout the U.S., LIRS provides training at client-designated sites for a 
fee. (Many public-access database producers also offer short courses periodi¬ 
cally in major cities in the U.S. but less extensively than LIRS). Although 
NLM had provided abbreviated introductory and advanced courses at Regional 
Medical Libraries (RML) at least quarterly, travel and personnel constraints 
may limit this in the future. NLM exerts a high degree of quality control 
over its courses. Through 1980, it had authorized only one institution, the 
RML at the University of California, Los Angeles, to provide training recognized 
as equivalent to that given by NLM staff. (NLM and UCLA trainees receive 
Continuing Education credits from the Medical Library Association.) One or 
more other institutions may receive NLM authorization in 1981. Some instruc¬ 
tors don't favor travel because of the added burden of fatigue on an already 
stressful and fatiguing task. Other instructors regret their inability to 
bring training out of the artificiality of a classroom to the terminals and 
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actual work environments of individuals where they think learning and retention 
might be greater. 

Undoubtedly some people who have wanted to attend courses that are given 

at only one or a few locations have been unable to do so because of travel 

expenses. Even when there is no course fee, as in six of the instances in 

this study, travel costs can range, conservatively, from $100 to $500 for 

transportation, $20 to $45 per day for hotel/motel accommodations, and $20 to 

$30 per day for food. The LIRS arrangement that brings one instructor to any 

location for $300 for a half day, $375 for 2 half-days, and $500 for 1^ days 

is more cost effective if (sometimes a big "if”) the training site is adequately 

equipped. The ONTAP mode of learning may be the most cost effective for many 

students when it has been extended to a larger number of databases. In a test 

of ONTAP CHEMNAME, university chemistry students needed 20 hours to complete a 

Chemical Abstracts Service (CAS)-produced course. Expenses included $20 for 

the CAS-prepared manual, $60 for 4 hours of connect time (at $ 15/connect-hour), 

and the costs of CAS's printed and microform search aids} the latter aids are 

4 

reference materials that can be shared by many students. LIRS gives 2 hours 
of free access to ONTAP files to those who attend the $65 introductory course. 

2a. PROGRAM PREPARATION COSTS 

Program preparation refers to all of the activities that are involved 
in determining the scope of an education program, the specific content of each 
segment, the sequence of the various segments, and the instructional materials 
and aids to be employed in each of the segments. To teach the use of online 
databases, a considerable amount of time is usually spent in developing 
illustrative search strategies and portions of search strategies to demonstrate 
particular features of a command language, one or more databases, or both ol 
these elements. The selection of appropriate strategies for the first time 
(for a new manual) can be particularly time consuming because file responses 
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may differ from expectations, necessitating new search strategy development. 
Instructors usually encounter some system down time during manual preparation 
that prolongs the material preparation time. 

The initial and update preparation times and costs presented in Table I 
are believed to be very conservative. The code "est." is used for data that 
were not directly provided but were inferred from other information that was 
supplied or from the investigator's estimate of times or salary levels. It 
is believed that the nature of program preparation tasks is such that, unless 
they are timed as they are performed, recall about times is far from accurate. 

An effort has been made to limit the tabulated times and costs to professional 
and technical work, specifically excluding typing and printing times and costs. 
The CRS contract costs were not subdivided and undoubtedly include overheads 
and clerical costs as well as professional costs. One might conservatively 
double costs of the other programs before comparing them with CRS costs 
(doubling CRSin-house costs as well). For Pitt and ONTAP, times and costs are 
given only for the production of the syllabus and printed instructional 
materials; the costs of the programming to produce Pitt's DIALOG and ORBI" 
simulators and LIRS's ONTAP software are not known. At Pitt, one gifted under¬ 
graduate computer science student was credited with writing and maintaining 
the simulators. 

The data suggest that program planning and development plus the produc¬ 
tion of reasonably comprehensive written manuals take about 1 to 1 2 months for 
each content focus, for example, instruction primarily about a command language 
(that introduces database content mainly to illustrate features of the language) 
or instruction about one database (that introduces elements of a command 
language to illustrate retrieval options from the database). When work is 
coordinated among several people, an additional^ to 1 month may be needed. 

When the instruction encompasses a command language and one or more databases. 


C- 
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the preparation time seems additive. For example, DTIC used 8 man-months lor 
a command language and 3 databases and Pitt used 6 man-months for DIALOG, 

ORBIT, and a general coverage of LIRS's and SDC's databases. When the data¬ 
bases are complex, the preparation time can be longer, e.g., 3 months for 
UDRI's Profiles course. When the command language and the databases are simple, 
some time compression can be achieved, e.g., \\ to 2 months for language plus 
database in H.I.S.'s system. 

Not unexpectedly, CA1/CBI requires more initial development time. 

ONTAP is only minimally a CAI program. By providing the student with 

- Number of citations in the answer set, 

- Number of citations found, and 

- Recall and precision percentages for search strategies 

the student creates for "canned" questions, 

the program helps the student learn how to achieve high recalls and precisions. 
Underlying the program is work by LIRS staff and 15 professional librarians 
to develop and refine 29 test questions and to arrive at a consensus about 
relevant citations retrievable from a test file of 32,120 ERIC records. 

CRS has used the services of a commercial firm, Control Data Corp. 

(CDC), to obtain a true CBI program. CDC markets PLATO (Programmed Logic 
and Automatic Teaching Operation), a system developed in the mid-1960's at 
the University of Illinois. Several other CBI systems are also on the 
market; for example, H.I.S. is currently experimenting with Boeing Computer 
Services' SCHOLAR/TEACH 3. The CRS program now in use is a rewrite. The 
first version was wordy, long, and required little student interaction. 

With redirection supplied by CRS staff, the revision is subdivided into 
modules, each of which can be completed in 8 to 15 minutes, and requires 
considerable intellectual participation (rather than just button pushing). 

The $100,000 cost, however, is ten times that of most of the other instruction 
programs and twice that estimated for the ONTAP ERIC program (ONTAP CHEMNAME 
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may be more costly than ONTAP ERIC). 

3. RESOURCES 

Student/Instructor ratio is one of the few criterion variables that 
is inherently quantitative. Does it have predictive value for course effect¬ 
iveness? The best answer is that this is not now known. In the DTIC, NLM, 
and Pitt situations, and partially for UDRI, the upper number of students 
is determined by available terminals for hands-on practice. Pitt enlarged its 
terminal capacity in early 1981 and is planning on experimenting with a 
larger class size. 

Only the Pitt, ONTAP, and CRS programs provide each student with a 
terminal. H.I.S. could readily redesign its format to a one person-one 
terminal mode but finds that trainees are more comfortable working in pairs. 
DTIC, NLM, and UDRI are less flexible in format redesign, but instructors 
indicate participant satisfaction with the 2 or 3 per terminal arrangement. 
Observation at Pitt and personal experience at GSLIM suggest that many people 
prefer the one-to-one arrangement if given the option after they have gained 
experience with the system and databases. Equipment and online costs that 
this arrangement could generate may prevent its use except when terminals 
are online to internal computer systems. 

The TI Silent 700 continues to be a popular terminal. Notably, NLM 
and Pitt terminals are not equipped with CRTs although the instructor's 
terminal at Pitt has a CRT and the training room has 3 TV monitors for online 
demonstrations. H.I.S. and CRS terminals have no printers. Most H.I.S. 
clients have single item questions for which they need to knew answers, but 
they have no need for a written record. Printers for them were said to be 
distracting. They are informed of a print capability to which their terminals 
are linked should a printout be desired. Users of CRS's PLATO program receive 
an instruction booklet that lists the retrieval capabilities of the databases. 
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this may be adequate or more printed materials may be found necessary with 
usage of the program. I found myself recording the content of screens in a 
notebook and recognizing the importance, after the terminal experience, of 
information I didn't record. H.I.S. and a segment of CRS's trainees differ 
from those of other programs. Except for some users of the CHEMNAME ONTAP 
program, most students of the other programs are, or expect to be, working 
as intermediaries for end-users. By contrast, most H.I.S. and CRS students 
are end-users. These end-users want to learn only enough to be able to 
retrieve needed information. 

Only GSLIM, Pitt, and NLM give evidence of having invested time and 
money in classroom design, even tho these facilities share with the others 
a starkness and overcrowding. LIRS and DTIC give courses in the field and 
are subject to available facilities that can have fewer terminals and closer 
quarters than their home classrooms. It appears to be a tribute to the topic 
being taught that so many people, instructors as well as students, have been 
willing to tolerate poor facilities with little complaint. 

Specially prepared manuals or less ambitious guides or booklets are 
produced in all of the programs. The sentiment was repeatedly expressed 
that printed materials that students could refer to on the job are an essential 
regardless of other media that may be employed. DTIC, NLM, UDRI, and Pitt 
have produced the most extensive manuals/workbooks. NLM's two Workbooks for 
its introductory and advanced courses are by far the most comprehensive. 

NLM's objective was to reduce students' need to take in-class notes to a 
minimum so that they could concentrate on the lecture content. DTIC and NLM 
also distribute other publications (e.g., thesauri, searcl aids) that are 
needed to learn their respective databases. LIRS, GSLIM, and Pitt include 
references to database thesauri, vocabularies, and search aids in their 
instruction; usage of these materials is part of GSLIM's and Pitt's instruc- 
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tion, but students must independently purchase these auxiliary materials. 

By contrast, the materials H.I.S. and CRS now provide are much shorter than 
the manuals that were initially distributed; they contain only database 
content and searching essentials. The content modification is based on 
client acceptance-testing. 

Most of the manuals/workbooks are instructor transferrable in the 
sense that instructors other than those who prepared the manuals could readily 
use them and provide instruction that was consistent in coverage and content. 
The brevity of LIRS materials requires a lesson plan auxiliary to the 
materials. To use UDRI materials, the instructor must have independent know¬ 
ledge of the FTD databases that have structural differences from bibliographic 
files and are complex. Individuals not privy to rationales behind CRS's 
PLATO programs may not be able to provide satisfactory assistance all the 
time to students who press the human HELP button. 

NLM and Pitt minimize material preparation costs and maximize currency 
of their materials through the use of an information technology. NLM*s work¬ 
books are produced with Vydec word processors. Pitt uses the VAX computer, 

NLM instructors review the text approximately every two months for currency. 
The Pitt review is approximately monthly. Both organizations print only the 
number of copies they expect to need for courses during the period. Thus, 
they have few outdated copies that need to be discarded or updated. Addi¬ 
tionally, both use the looseleaf format so that last-minute insertions can 
easily be made and to enable students to amend their texts or insert auxiliary 
materials. 

4. EVALUATION/FEEDBACK MECHANISMS 

Minimally, two parties in an education situation want a measure of 
course effectiveness. The student wants to know whether he/she has learned 
what should have been learned; this relates to course objectives and is one 


important reason [or performance-measurable objectives. The instructor wants 
to know whether the content, the method of presentation, time allocations to 
various topics, etc., achieved the intended information transfer. Frequently, 
host institutions and sponsors want students' perceptions of the adequacy of 
the instructor(s), course content, and/or the study facility/equipment/ 
computer system, etc. All parties are interested in effectiveness -- how 
well the program achieved its goals. 

The most prevalent instrument that has been used for evaluation in 
education is the written test administered to the student. Except for the 
university course, no such test that is submitted to the instructor for grading 
is used in online instruction. All but the LIRS courses give students various 
degrees of opportunity to assess their own progress through the assignment of 
problem queries for which students individually or in groups develop search 
strategies that they test in online practice. Most instructors view this 
mode of instruction as essential to retentive learning. It has no substitute 
for building student confidence and competence. NLM's MEDLEARN most resembles 
an academic test but it is self-administered. It is a CBI program that 
students accepted for NLM*s introductory course are expected to complete before 
arriving at NLM or UCLA. MEDLEARN is primarily expected to give students 
experience in logon/logoff procedures and some familiarity with online data¬ 
bases. Students receive special passwords and free time for MEDLEARN and 
can obtain telephone assistance if needed. 

Six of the programs distribute questionnaires to students at the end 
of the course that request evaluative information. The instruments for DTIC, 
LIRS, NLM, UDRI, and Pitt are presented in Appendix B. For additional compari¬ 
son, forms are also included that are used by the American Society for Infor¬ 
mation Science (ASIS) for short courses and preconference tutorials and by 
the U.S. Government for training course programs (Optional Form 170). DTIC's 
questionnaire and those of ASIS and the Government are the most formal. 
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Several of UDRI's questions are the most specific. Questions 3 and 4 
of the "Basic Class Evaluation" and the first page of "Online Profile Evalua¬ 
tion" questions, for example, probe details of the command language and 
search features. DT1C's questionnaire is at about the degree of specificity 
of the remaining questionnaires. The 5-level rating scale used in the ASIS 
questionnaire is often considered preferable because people usually perceive 
a finer gradation than the three options DTIC's questionnaire permits. The 
ASIS questionnaire seeks evaluative information at a slightly more detailed 
level than DTIC's, and in some instances the wording is more "user friendly." 

To the teacher, an unsatisfying aspect of teaching oftei. is the 
absence of a longer term feedback mechanism. L1RS, DTIC, NLM, and occasionally 
H.I.S. instructors get such information, though not directly and irregularly, 
from persons at the customer service desks of these organizations. The NLM 
arrangement probably provides the most feedback because members oi the MEDLARS 
Management Section perform all Section activities including manning the service 
desk and teaching. This distribution of tasks may change in 1981, teaching 
being assigned to 2 or 3 persons, but the physical proximity of Section members 
is likely to retain the feedback flow. 

User group meetings provide another mechanism for feedback. User groups 
have developed among DTIC and NLM clienteles. Each of the groups has regional 
quarterly meetings and an annual meeting at the central location \OilC and 
NLM, respectively). The Update programs sponsored by LIRS several times 
annually, usually betore or after the annual meeting of a major professional 
society concerned with online databases, are similar to user group meetings. 

The meetings are effective two-way communication mechanisms. The database 
producer or vendor disseminates new system information and "helpful hints" 
type material anil database users share experiences and discuss operating prob¬ 
lems. Suggestions presented at these meetings have effected search system 
changes as well as changes in instruction programs. 
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TABLE I. CHARACTERISTICS OF ONLINE INSTRUCTION PROGRAMS (cont 



(1) Purchase/Rental 
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TA3LE I. CHARACTERISTICS OF ONLINE INSTRUCTION PROGRAMS (cont 














































Usefulness Probably positive I N/A I Positive Positive H/A N/A 





































































CRITIQUE OF THE PROGRAMS 


One program can't be considered better or worse than another because of 
the differences among them. Implicitly or explicitly, they have different 
training purposes. They address different command languages and different 
databases. Often they are directed at different target groups. Experience 
from the programs, however, does yield guidelines that can be generally helpful 
for future program development. 

It appears that two essential elements of any instruction program are 
printed materials and hands-on practice. User need for manual specificity, 
depth, and written analysis can vary widely. H.I.S. and CRS experience 
should be a reminder that brief is beautiful for some user groups. A serious 
problem arises in a situation like UDRI's where database complexity dictates 
a comprehensive manual but students lack experience to cope with such a text. 

The question "What is enough hands-on time?" is not an issue in the ONTAP and 
CRS programs (if money isn't a problem for ONTAP students) because the 
programs are self-paced. Pitt believes that the 10 to 12 hours of practice 
its students receive, accompanied by instructor assistance as needed, enables 
students to perform independently in their own work environments. GSLIM, 

DTIC, and NLM experiences indicate that 2 hours per student or 2-student 
group is not adequate if database learning is involved; 4 to 5 hours may be on 
the borderline of adequacy. 

It is fashionable to frown on lectures as an antiquated mode of pre¬ 
sentation. Its pervasiveness is probably partially attributable to instructors' 
lack of familiarity with other media, but also partially because of the 
additional investment of time (and hence cost) required to incorporate 
slides or videotapes, tor example, into instruction programs. The objective 
to be accomplished and the audience must be kept in mind. CRS's 14-minute 
introductory videotape doesn't have the sophistication of a CBS documentary 

•i 
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but it is informative, holds attention, and accomplishes the objective of 
introducing the viewer to SCORPIO databases. Videotapes could communicate the 
search process and system responses much better than lectures. The use of 
instructional television (ITV) is widespread and is likely to be as effective 
in teaching the use of online databases as it is in other areas. For content 
subject to change and when portability (e.g., borrowing by students, exchange 
with other institutions) is desired, tape/slide modules could be developed. 

A comprehensive British evaluation ot tape/slide guides for library instruc¬ 
tion recommends this methodology as low in cost, easy to use in program 
preparation, easy to operate, and robust in handling. Sound tapes alone 
are being tested by H.I.S. for self-paced learning; listening is integrated 
with a printed manual and hands-on practice. Slides are being effectively 
used by UDRI to reinforce or illustrate lecture content. The slides range 
from prose to tables to line drawings and make good use of color and different 
type sizes. In all instances where a technology is used, it is not used in 
isolation. As is suggested in the British study, "it may be that interaction 
between tutor and student, but in a structured setting, is still the most 
important ingredient." The lecture isn't obsolete, but an online instruction 
program that relies on lecture alone for more than 30-35% of the contact time 
with students probably needs to be rethought. 

Although the computer has had the potential to individualize instruction 
since the mid-195C's, and although CAI/CBI/CMI systems have been developed, 
utilization is still limited.^ A tabular summary of the major systems currently 
available to course writers is given in Table II (page 78). In an excellent 
review of the CAI state-of-the-art, Jurg Nievergelt cautions that more skill 
is required to use the medium effectively than is required by any other 

g 

medium. He estimates that an author must plan to spend between 10 and 100 
hours to produce one hour of student contact time, the realistic average 
for several authors working collaboratively being 100 hours. He anticipates 
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TABLE II. MAJOR CAI/CBI AND CMI SYSTEMS IN CURRENT USE" 


[- 

tYPE Of 


PROJECT 

DEVELOPER 

TYPE 

DESCRIPTION 

COMPONENTS 

SERVICES 

STUDENT 

CURRICULUM 

In- 

itflKUOAi 

m-hmiwi System 

(AIMS) 

New York Institute 
of Technology 

CMI 

Uses any course 
materials thilgned 
for behavior* 
objectives 

objectives, stu¬ 
dents, uu mutism, 
and faciUtios. 
instruct mnaf 
strategies. tom, 
and feedb—R 

Evaluation of 
student program, 
prater muons, 
and empirical vali¬ 
dation and optmti- 
tation of instruction 

Any 

Math 

Whconun System of 
Instructional Miiugr 
ment (WIS-SIM) designed 
foe Individually Guided 
Instruction (tG£) 

Wisconsin Research 
and Development 

Center for Cognitive 
Learning 

CM* 

Utilizes spedfk 
instructional ob 
teclives throu* 
instructional pro¬ 
gramming mod* for 
individual uudepl, 
establishes mastery 
levels 

Instructional pm 
pammmg mod*, 
measurement and 
evaluation mod*, 
and multi-unit 
school 

O tier ion-referenced 
tests, achievement 
profimg. diagnosis, 
prescription, and in¬ 
struction 

Elementary 

Reading, 

Math, 

Science 

Individually h»- 
scribed ? rructionf 
Management hdownU— 
System (IPI/MIS) 

Learning Research 
and Develop men! 

Center at University 
of Pittsburgh fitted 
by Research for 

Seiler Schools 

CMI 

Uses IP! curricular 
materials which cm* 

Ploy detailed m 
struct tonal objec¬ 
tives, methods, and 
materials, estab¬ 
lishes mastery level 

Detailed object¬ 
ives. methods, and 
materials, deter¬ 
mination of in- 
dhriduai compe¬ 
tence, continuous 
evaluation, and 
student perfor¬ 
mance monitoring 

Diagnoses and 
prescribes. Collects 
and proccsees infor¬ 
mation, competence, 
performance, and pro¬ 
cess of each student 

Elementary 

Mach 

leuiucwwd Mwy 

ment System (IMS) 

System Development 

Corporation South* 
west Regional Lab 

CMI 

Adapts to any cm 
Kulum which has 
dear, specific 
learning objectives 

Frequent testing, 
profiting, m* 
vidua! study recom¬ 
mendations, and 
summary reports 
on all tests 

Assnls with pacing, 
grouping, sequencing, 
and individualization 

Any 

Readtng. 

Math, 

Drama. 

Pubik 

Speaking. 

Art. Musk 

^rofM for Learning 
in Accordance writ* 

Nee* (PLAN*) 

American Institutes 
for Research and 
now managed by 

HIM 

nmnifmoi 

Learning Corporation 

CMI 

Employs prescribed 
behavioral objec¬ 
tives and Teaching 
Learning Units 
(TLUs) 

Behavior* objec¬ 
tives, Teaching- 
Learning Units, 
enter ion-refer¬ 
enced tests, gui¬ 
dance, indi¬ 
vidual planning 
and teacher dne- 

Monitoring and 
supervising, test 
scoring, diagnosis, 
presetipiions, in- 
dividualitaiion, in- 
service for teachers 

Any 

m 

Util ties ex¬ 
isting mater¬ 
ials m Language 
Arts. Social 
Studies, Math, 
Sciences 

Inter active Trade- 
•nf System (ITS) 

International 

Business Mtchbtet 
Corporation 

CAI 

Allows teacher to 
write course with¬ 
out requiring de¬ 
tailed programming 
knoehrfgi 

Course structuring 
feature, author 
plans and wrttm 
course. Undent 
sees ions may bo 
scheduled or lb 
kon ai random, 

to answer questions 

Develops courses, 
teachers write cour¬ 
ses, any mode of pre¬ 
sentations is permit- 

led 

Any 

Any cosme 

Programmed Lofk for 
Automatic Teaching 
Operation (PLATO) 

University of 

Illinois, now 
managed by Control 
Oata Corporation 

CAJ 

In ferrates knewledgi, 
fact, information, 
and education devel¬ 
oping systems throw#! 
audio and visual 
means 

Teacher-designed 
lessons, evaluates 
and monitors stu¬ 
dent performance 

Any CAI mode can be 
employed, develops 
course* and units, 
especially helpful 
with simulations and 
game playing, revision 
and editing at any time 

Any 

Utilises a 
wide range 
of subtree 
matter formats 

Stanford Project 

Stanford 

University 

CAI 

Use* prop ame and 
materials with struc¬ 
tured curriculum 

Lemons found in 
materials, teacher- 
developed, monitory 
and evaluates 
student performance 

Revisions and editing 
at any time, but for 
problem solving and 
drill-praetke, schools 
usually contract for 
servlcaa 

Any 

Applicable to 
reading, arith¬ 
metic, logic. 
algebra, problem¬ 
solving 

Time-Shared, Inter¬ 
active, Coapwer-Coo 
t/oNed, Information 
TekvHkm (TKXlT) 

MITRE Corporation 

CAI 

U titties aH four 
mod* of CAI tit 
provide besom for 
students and •*- 

HzetmedU 

Design lemons 
for individuals, 
employs ag lour 

CAI modes, eval¬ 
uates and moni¬ 
tors student per* 

Revision and editing 
of program, monitor¬ 
ing, and evaluating 
student progress 
utilising aH four 
modes of CAI 

Any 

Utilizes a wide 
range of oust¬ 
ing subject 
matter formats 


lormiM 


Excerpted from: Fred L. Sp1ittgerber, "Computer-Based Instruction: A Revolution 
in the Making?" Educational Technology, 19:1 (Jan. 1979) 20-26. 
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that the cost of GAI/CBI programs will decline as CAI/CBI becomes a mass 
consumption item at which time the cost may be no greater than that o£ ITV. 

CAI programs are presently limited in the assistance they provide to student 
questions. In Nievergelt's words, "the limitations of what you can program 
show up starkly and painfully." CRS' s PLATO program covers the basic elements 
of the SCORPIO language and the contents and retrieval options of four data¬ 
bases straightforwardly and in an interactive way with frequent questions 
the student must answer that holds student interest. Branching limitations 
are quickly evident in HELP responses. Sophisticated applications o£ SCORPIO 
and complex search strategy possibilities are not included in the programs. 

CRS has demonstrated that CAI/CBI can be successfully used for an introductory 
online instruction course but it is not likely to be realistically applicable 
to advanced content. 

At probably half the cost of CAI/CBI programs, the ONTaP methodology, 
particularly as implemented by CAS, may be a viable approach to giving students 
who plan to be or are searchers a range of experience from the introductory 
to the advanced level. Although the 16 user questionnaires that were obtained 
are too few for definitive analysis, more hands-on practice and more search 
strategies/case studies/prototype searches appear clearly as desiderata (see 
Appendix D). Po’-haps other database producers c.an be encouraged to follow 
the CAS lead. On he assumption that a $15/connect-hour cost is quite reason¬ 
able, would-be students may have to sell employers on the cost effectiveness 
of this mode of learning. 
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I IMPLICATIONS FOR DTIC INSTRUCTION PROGRAMS 

This section addresses instruction program design features that DTIC 
should consider in formulating plans for future programs. The program elements 
are discussed in the order used eirlier, namely 

1. Purposive documentation 

2. Education program 

3. Resources 

4. Evaluation/feedback mechanisms 
1. Purposive Documentation 

DTIC should provide a clearer statement of basic program goals for its 
introductory and advanced courses than exist at present. The existence of 
these separate courses, augmented by several abbreviated formats, indicates 
that different end results are expected. 

The NLM practice of specifying behavioral outcomes expected of students 
on a lecture-by-lecture basis is initially laborious to do, but the result is 
beneficial for both instructors and students. Course designers and instructors 
have a clear, unambiguous direction that proves easier to work with for course 
development than if it were not present. By phrasing objectives in performance 
measurement terms and sharing them with students, the students know clearly 
what they must achieve. The documentation also permits all parties to the 
instruction program, including external critics and other organizations that 
may wish to provide instruction, to review and evaluate it and, perhaps, make 
constructive suggestions for improvement. 

Criteria -- particular outcomes designers want to achieve -- are 
developed in conjunction with behavioral objectives. Minimally, they should 
answer the question: How much of the performance attainment must be attained? 
Where quantities can be specified, such as student/instructor ratio, number 
of hands-on practice hours, number of search strategies to be formulated, 
single numbers or a numeric range should be specified. An important facet of 
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criteria development is that the criteria accurately characterize course 
desiderata. Factors that are measureable but are not important to the goal 
of the course or are not descriptive of student achievement have small value 
and should not be employed. 

2. Education Program 

The multi-course structure is a product of both wisdom and necessity. 
Except for the academic environment, students in other situations are usually 
employed. They either cannot, or believe that they cannot, be away from their 
work for more than one week at a time. Additionally, short courses almost 
always are intensive, requiring 7 or 8 hours of continuous concentration per 
day. Pitt's course includes two 2 or 3 hour evening sessions as well. People 
become fatigued in these situations even when the presentation is varied 
through the use of different media and hands-on practice. Strain among 
students often appears after to 3 days, and the effect on learning and 
retention during a longer period, while not known, may be serious. Thus, 
the course subdivision that DTIG now has should be continued. 

The content of DTIC's two regular courses and their modifications 
bears re-examination. The success LIRS has found with modularization and 
NLM's experience that will probably result in optional modules at the advanced 
level appears applicable to DTIC, particularly since public access is intro¬ 
ducing clients with different, less subject intensive, needs than those of the 
more homogeneous DOD community. An introductory course consisting of a 4-day 
format is suggested to cover the fundamentals of the command language, the 
TR and CF report files, the WU file, and the PP file. The overall objective 
of the course could be to provide a functional understanding of, and experience 
with, the databases to enable students to do query analysis and online retrieval 
at their work stations. 

DTIC should then consider the development of advanced modules that 
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focus (a) on the command language and (b) on each of, or groups of, the 
databases. The modules could be 1 or 2 days in length as needed. A module 
is also needed for the Independent Research database. These modules could 
be offered back-to-back over a one-week period or, based on client demand, 
some could be offered more frequently than others. The module approach is 
suggested to more closely accommodate to client needs and to indirectly assure 
the attentiveness of students. DTIC might wish to institute a requirement 
that participants for advanced modules have 3 or 6 months or some other period 
of database and system experience prior to enrollment. DTIC might also wish 
to prepare a self-test by which those wishing to attend advanced modules 
could determine their readiness for the modules. 

Cutbacks in travel expenses can be expected to more seriously impair 
the ability of people who should receive instruction from attending courses 
conducted in one or a few locations. On the assumption that instruction pro¬ 
grams will continue to be lecture based for at least the next few years, the 
number of sites at which the instruction program is made available should be 
appreciably increased. Locations could be announced with an understanding 
that assurances of attendance below a certain number by a certain date would 
cancel the course. 

A trend for the past several years has been the development of education 
programs by database producers. FTD has elected a different option, namely 
contracting for instruction program services. Since the DTIC databases are 
not as complex as several of FTD's, the problem of ensuring full contractor 
understanding of the database structures and contents and characteristics 
of the client communities is less severe. The contract option nevertheless 
requires that the contractor have a thorough knowledge of the DTIC system 
and that a close working relationship develop between contract and DTIC 


personnel so that instructors can keep apprised of system changes 





The issue of an instruction program fee becomes increasingly important 
if instruction is to be given extensively in the field. Insofar as DTIC is 
a federal agency that wants the public to utilize its databases, a high fee 
seems inappropriate. However, people tend to place little value on what they 
get without cost. Therefore a modest fee of $50 or $60 for the introductory 
course and $30 for a 2-day module seems reasonable, particularly if locational 
dispersion reduces student travel costs and the fee includes a workbook or 
manual. 

3. Resources 

Instructional experience and student feedback strongly suggest that 
all instruction programs incorporate online hands-on practice. The weight of 
evidence points to sufficient terminals so that each participant can have a 
terminal or, as a minimum, a terminal can be shared by no more than 2 persons. 
Some people are terminal shy and, even in 2 or 3 person groups, they avoid 
practice. This may prevent others from seeing their slowness or mistakes, 
but this does them a longer term disservice because they may not be able to 
function when they return to their workplaces. A $600 or $800 or $1000 invest 
ment per terminal that may be used by 20 to 30 to 40 people per year in a 
classroom reduces the per-student investment to nominal amounts. Arrangements 
can be made to rent terminals at field locations. Most students seem to inter 
act better with terminals having CRTs, but since many work-station terminals 
do not have CRTs, a mixture would be more economical and may be more realis¬ 
tically instructive to students. 

The optimum student/instructor ratio is not yet known. Experience 
shows that 20/1 does not permit the instructor to meet students' needs. A 
3/1 or 4/1 ratio appears to make more instructor time available than students 
need during most of a course (though not at the very beginning). A 10/1 or 
12/1 ratio may be the most cost effective for DTIC on the assumption that most 
trainees are already users of an online system. They are likely to find the 





DTIC command language a little strange because it differs from the one they 
already know, but they also know what they must learn. The DTIC databases are 
sufficiently similar in structure to other bibliographic databases as to again 
permit most students to learn through self-teaching. In such an environment, 
one instructor should be able to attend to all of the questions posed by 10 to 
12 students. This is another element of an instruction program where spreading 
a resource too thinly may quantitatively indicate cost effectiveness but it is 
likely to impact the objective of producing self-reliant system users. 

Instructors uniformly agree on the value of printed tutorial materials 
regardless of the instructional mode used for knowledge transfer. Experience 
indicates that different formats are acceptable to different audiences. As 
other agencies have done, DTIC has found it advisable to produce both long and 
short versions of manuals, i.e., the "Self-Training Module for Users of the 
Defense RDT&E On-Line System" dated September 1979, a companion volume for 
dialup clients dated October 1980, and the "Retrieval Mini-Manual" dated 
December 1979 as well as a quick reference booklet, "Defense RDT&E Diverse 
Dial-Up On-Line System Reference Guide" dated October 1980. All three levels 
of description and explanation are likely to be needed in the future. 

As has been noted previously, DTIC's future dialup clientele can be 
expected to be much more heterogeneous than it was when access to DTIC's 
databases was limited to DOD agencies and contractors via dedicated terminals. 
Some clients will probably have little knowledge of technical information 
systems and will want to be able to use a few basic commands for limited 
access to the system. A Mini-Manual type of training and reference text will 
probably meet their needs as well as a need for quick verification of a command 
or a code by more sophisticated searchers. Information professionals performing 
as search intermediaries usually want as much detailed illustration of search 
strategies and search alternatives as a database producer or vendor is willing 
to supply. This kind of text could be written so as to be employed in 
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conjunction with course delivery as NLM does with its Workbooks. Alternatively, 
the detailed manual could contain material augmenting that covered during the 
course and possibly serve an encyclopedic function. 

Use of a variant of NLM's and Pitt’s mode of producing printed materials 
is highly recommended. The ubiquitous word processor was initially designed 
for this type of production task. 

The nature of the course material suggests study facilities that provide 
adequate table or desk space for the various manuals, thesauri, search aids, 
and notebooks that must be examined, learned, and used during the course of 
the instruction program. The amount of surface provided for students in DTIC's 
current classroom requires that students stack documents on top of each other 
and use the floor and their laps for overflow. Furthermore, chairs are less 
than comfortable for long sitting periods and are particularly skimpy for 
large men and women. DTIC's classroom is a good example of an excessively 
cramped facility. The instructor has little room to move about for chalkboard 
or chart use. If slides or transparencies were to be introduced into lectures, 
the equipment would further reduce student work space as well as introduce wires 
likely to be tripped over. Functional designs, furniture, and layouts are 
being developed by manufacturers of office and education equipment. Whether 
or not their products are bought, they can be examined for ideas that could be 
incorporated into study facilities that should be viewed as learning labora¬ 
tories. 

The lecture mode of program delivery presently in use needs at least 
some integration with other media. (See Appendix C that presents advantages 
and disadvantages of several instruction methodologies.) As a minimum, 
command language operations, search strategies, and system responses could be 
exemplified on slides or transparencies. Students should have copies of these 


visuals as separates or in a workbook for note taking. DTIC could experiment 
with the development of tape/slide modules or TV film for instruction about 
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DTIC's databases. These could be used instead of lectures or could be made 
available for student borrowing as refreshers or in advance of lectures, 
thereby using the face-to-face time with an instructor for question/answer 
discussion. Computer systems often go down during instruction when they are 
most needed. It is suggested that backup material be developed on slides, 
transparencies, or film that can adequately replace all online demonstrations 
should the need arise. Alternative student work activities must also be 
developed as substitutes for online practice sabotaged by down time. 

CAI programming is not recommended for DTIC at this time because (a) 
it is too expensive to develop for limited numbers of users and (b) it is 
not at the off-the-shelf level yet for advanced users. The ONTAP program 
does warrant investigation by DTIC. It is suggested that DTIC explore the 
activities undertaken by the Chemical Abstracts Service to implement ONTAP 
CHEMNAME. It is believed that many information professionals who will be 
using DTIC databases and who need a DTIC refresher, as well as end-users in 
DOD, are likely to find an ONTAP-like program useful. Implementation of such 
an instruction program could alleviate the need for, and expense of, field 
instruction programs. 

4. Evaluation/Feedback Mechanisms 

DTIC is already using two important evaluation mechanisms, a course 
evaluation form completed by students and periodic user group meetings. 

Although it might be desired that the evaluation form more adequately 
provide for written comments, it seems sufficiently adequate to retain. The 
main problem with an evaluation instrument administered immediately after a 
course is that students have not yet had the opportunity to apply what they 
have learned in a real situation. Evaluation would be much more useful if it 
could be obtained after the student has used the system for 4 to 6 months. 
Therefore, it is recommended that an evaluation instrument be developed with 






questions similar in format to those of ASIS and adequate space for written 
comment. User group meetings may be the best forums at which to distribute 
(and collect) the forms. The form could also be a periodic insert in DTIC's 
newsletter with a short article promoting the evaluation. 

DTIC might also wish to adopt a pre-test mechanism for attendees of 
advanced courses similar to that of NLM. The pre-test is mailed to persons 
wishing to attend one or more of the advanced modules. It is a means by which 
potential students can judge whether they have acquired enough skill with the 
system to qualify for the advanced class, and it can identify areas of weakness. 
The tests are not sent to NLM for grading so they do not add to the instruc¬ 
tor' s workload. 

DTIC could systematize evaluative information acquired at its customer 
service desk to feed back user problems and experience to instructors, whether 
instructors are DTIC or contract personnel. Ideally, it is believed that 
instructors could benefit from spending even a small amount of time per month 
in a customer service capacity. This is the least artificial means of obtaining 


feedback and evaluative information and it should be utilized 
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INTERVIEW SCHEDULE 


INFORMATION SOUGHT ABOUT PROGRAMS/COURSES THAT TEACH ONLINE USE OF 
COMPUTER-STORED DATABASES, FILES 

1. What types of courses are offered? (beginning, intermediary, advanced) 

2. What are the durations of the courses? 

3. When are they given? (dates, particular times of the year) 

4. What are costs of the courses to participants? 

5. Are courses subsidized? By whom? By what percentage and dollar amount? 

6. What are the purposes of the courses? 

(what level of competency is it intended to provide?) 

7. What class size is preferred? What is the preferable class size limit? 

8. What course materials are used? Who prepared them? 

9. What teaching methodologies are used? 

(lecture, discussion, transparencies, programmed texts, CAI, 
hands-on, etc.) 

10. What is the size, composition (background) of the teaching staff? 

11. What are the typical education, experiential backgrounds of students? 

12. What information content is taught? 

13. How many hours / What proportion of time is allocated to hands-on 

learning? 

14. What terminals are used? 

(keyboard only, CRT + keyboard, whose manufacture, models) 

15. How many students are assigned per terminal during hands-on sessions? 

16. What kinds of assignments are performed during hands-on sessions? 

(drill & practice, search queries, parts of queries, independent 
project, etc.) 

17. Is a formal student evaluation performed? 

(copy of evaluation instrument) 

18. Is there a formal feedback mechanism some time after (6-12 mo.) the 

program/course is completed? 

(copy of feedback instrument) 

19. Particularly if 17, 18 not done, how is program/course effectiveness 

assessed? 

20. What assistance is given in addition to courses (i.e., after courses) 

to help with problems students encounter in the use of databases on 
the job? 

21. What program changes/improvements are being considered/planned/ 

desired? Why? 

RWS 9/80 







.ON-LINE EDUCATION QUESTIONNAIRE 


I. Whose programs for teaching the online use of databases have you sttended? 


Please Indicate vhen you had the education/training by narking 
the appropriate boxes. Please note ALL programs that you have had. 


Program 

Approximately how long ago did you have the program? | 

In 1980 

In 1979 

In 1978 

In 1977 

In 1976 

Wliir 

1, BRS - Introductory 







2. BRS - Advanced 







3. Lockheed - Introductory 







4. Lockheed • Advanced 







5. SDC - Introductory 







6. SDC • Advanced 







7. HLM - MEDLINE 







B. LEXIS 







9 , OCLC 







10. DT1C (formerly DDC) 







11. DOE/RECON 







12. NASA/RECON 







13. CIST1 - CAN/OLE 







14. Cuedre Associates 







15. On-the-job, forme1 







16. On-the-job, informal 







17. University - short 
course (1 week) 



— 




18. University - short 
course (2 weeks) 







19. University - regular 
course (qtr, semestr) 







20. Other (pis specify) 







21. Other (pis specify) 







22. Other (pie specify) 








2. On a scale of 1 to 10, please rate the programs you have had on how well 
they prepared you to do online searching/retrieval, by circling the 
appropriate rating number below for each program. 

"1" signifies very inadequate preparation 

"10" signifies preparation that permitted you to do searches 

on your own with a high degree of confidence and proficiency 

Space is provided below for ip to 8 programs. Please use the numbers 


given in the 

left colusm of Question 

1 on 

the 

lines 

below 

• 





Inadequate 







Adequate 

Program . 

1 

2 

3 

4 

5 

6 

7 

B 

9 

10 

Program . 

1 

2 

3 

4 

5 

6 

7 

6 

9 

10 

Program . 

1 

2 

3 

4 

5 

6 

7 

B 

9 

10 

Program . 

1 

2 

3 

4 

5 

6 

7 

6 

9 

10 

Program _ 

1 

2 

3 

4 

5 

6 

7 

B 

9 

10 

Program . 

1 

2 

3 

4 

3 

6 

7 

B 

9 

10 

Program _ 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

Program . 

1 

2 

3 

4 

5 

6 

7 

6 

9 

10 


3. What teaching/presentation methodologies do you like? Dislike? 

(e.g., lecture, films, transparencies, tape-slide, role pitying, discussion) 
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4. Whst aspects of Cbe content* of the programs did you find particularly helpful? 


5. What program content did you consider NOT adequate? (l.e., you wanted wore of) 


6. What content was NOT included that you would have wanted? 


7. Bow many class hours or workshop days do you think are needed to gain a 
working knowledge of 

A single database? ____________ Two related databases?_ 


8, Which of the follcMlng alternatives do you consider the best to acquire 
searching proficiency? (Please circle the appropriate letter) 

a. A single course followed by on-the-job practice 

b. A beginning course, on-the-job practice; 6-12 months later, an advanced course 

C. A beginning course, on-the-job practice; 6-12 months later, an advanced 

course on the c o mmand language and system and advanced database courses 
d. Alternative a and periodic 1-day updates on system + databases 

s. Alternative b and periodic 1-day updates on system + databases 

f. Alternative c and periodic 1-day updates on system ♦ databases 


9 . 


Which organisations do you think are scat able to provide high-quality education/ 
training In the online uae of databases? 


Boor 


Database producers 1 2 
Computer system vendors t 2 
Academic - library schoola X 2 
Academic • adult education service X 2 
Library netvork/servlce organisations 1 2 
Consultants, information broker* \ t 
Other (please specify) 1 t 


Cood 
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10. What kinds of latching materials do you prefer? 


11. Are you likely to buy a self-teaching annual Instead of attending a program? 

. Yes . - Ho Maybe 

12. Are you likely to pay for a self-paced cosputer-asslsted Instruction program 
Instead of attending a more traditional classroom program? 

_Yes _Mo _Maybe 

13. Bov ouch online searching do you do? 

Please circle the appropriate letter below. 

a. Over 20 hre/vk d. 5-10 hre/wk 

b. 15-20 hra/wk e. 3-5 hrs/vk 

c. 10-15 hrs/vk f. %-3 hre/vk 

g. don't do searching 

14. Bow frequently do you use the vendor systems listed below? 

Please note ALL that apply 



4 or more 
hrs/day 

2-4 

hrs/day 

0-2 

hrs/day 

approx 6-8 
hrs/vk 

4-6 

hrs/vk 

2-4 

hrs/vk 

occasion¬ 

ally 

BRS 








Lockheed 








SDC 








OCLC 








MEDLINE 








N Y Times 








NASA/RECON 








DOE/RECON 








LEXIS 








WESTLAW 








CAN/OLE 








DTIC 








In-house files 








--- |- 

Other (*pecify) 








15. Vhat type of organisation do you work foil 

a. For profit (not a DOD contractor) 

b. For profit (DOD contractor) 

c. For profit (lnforoatlon broker) 

d. Hot for profit (non-academic) 
a. Hot for profit (academic) 


f. Federal govt, (non-mllltery) 

g. Federal govt, (military) 

h. State, local government 
1. Self esgsloyed 


16. Do you have any other consents, suggestions that could help designers and 
developers of future education/training programs? 


Please return this Questionnaire to: Sevens Swanson / 6011 Shaffer Drive / 
Alexandria, Va. 22310 ^ thank YOU FOR YOUR ASSISTANCE Hi 


10/80 
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APPENDIX B 

COURSE EVALUATION QUESTIONNAIRES 


Lockheed Information Retrieval Service ..... 95 

Defense Technical Information Center ....... 98 

National Library of Medicine . 100 

University of Dayton Research Institute .... 104 

University of Pittsburgh . 112 

American Society for Information Science ... 114 

United States Government .. 116 
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DIALOG SYSTEM SEMINAR 


Name__ Phone 

Title/Oivision__ User No. 

Af-Filiatior.'Cc.____________ 


Whe r e did you learn about Lockheed DIALOG? 

Professional Meetings ( ) Advertisements ( ) Colleagues { ) 
Other (specify) _ 


Pleas? indicate below your experience level with online information retrieval 
systems. 


DIALOG 0 

to 

10 

hrs. 

10 

to 

25 

hrs. 

over 

25 

hrs 

0 

to 

10 

hrs. 

10 

to 

25 

hrs. 

over 

25 

hrs 

0 

to 

10 

hrs. 

10 

to 

25 

hrs. 

over 

2E 

hrs 

0 

to 

10 

hrs. __ 

_ 10 

to 

25 

hrs. 

_ over 

25 

hrs 


What subject areas are most important to you? 


Bi ored ici 

n 

Food/Agriculture 

( ) 

Physical Sciences/ 

Business 

( ) 

Government 

( ) 

Engineering ( ) 

r ne*,i $ try 

( ) 

Humanities 

( ) 

Social Science ( ) 

Education 

.( ) 

Patents 

( ) 


Other 






What databases would you like to see added to the DIALOG system? 





(DIALOG, LOCKHEED 


INFORMATION RETRIEVAL SERVICE, pg. 2) 


Training Location __ 

Instructor __Date _ 

How would you rate the content of this session on a scale of 1 to 1C? 


needs 

improvement 

satisfactory 

good 


excellen 

1 2 

3 4 5 6 

7 

8 

9 10 

How would you ra 

te the presentation? 




needs 

improvement 

satisfactory 

good 


excelle r 

1 2 

3 4 5 6 

7 

8 

9 10 

What portion(s) 

of the agenda did you find 

most 

useful ? 



What suggestions would you make for improving the session? 


Additiona 1 comnents or, the training session? 


Additional comments or suggestions for the DIALOG retrieval service? 


We appreciate your consents, and read them all. They play an "important 
in developing training to suit your needs and in continually 
imi'-oving the DIALOG service. Thank you for your time. 




(LOCKHEED INFORMATION RETRIEVAL SERVICE) 
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DIALOG Business Training Session 


The DIALOG Customer Services Staff is attempting to be responsive to the training 
needs of users. Your comments below will assist us in this endeavor. 

1. Name: __ DIALOG Instructor: _ 

Organization: __ City: _ 

Phone No. __ Date: _ 

User No. 


2. Please indicate below your experience level with online information retrieval systems 



DIALOG 


0 to 10 Hrs. 

10 

to 

25 Hrs. 


over 25 Hrs. 




0 to 10 Hrs. 

10 

to 

25 Hrs. 


over 25 Hrs. 




0 to 10 Hrs. 

10 

to 

25 Hrs. 


over 25 Hrs. 




0 to 10 Hrs. 

_ 10 

to 

25 Hrs. 


over 25 Hrs. 

3. 

Please indicate 

your 

feeling regarding the 

level 

of 

today' 

s session 

by circling 


a number. 









1 2 

3 

A 5 

6 


7 

8 

9 10 


Too 


About the 





_ Too 


Elementary 


Correct Level 




Advanced 

A. 

Please indicate 

your 

feeling regarding the 

pace 

of 

today's 

session by circling a 


number. 









1 2 

3 

A 5 

6 


7 

8 

9 10 


Delivery 


About the 





Delivery 


Too Slow 


Right Pace 





Too Fast 

5. 

How would you rate the presentation? 







1 2 

3 

A 5 

6 


7 

8 

9 10 


Needs 


Satisfactory 





_ Excellent 


Improvement 









6. Would you be interested in attending any DIALOG Training Sessions on other topics? 
These would be similar to today's session in that they would be one-half day, 
would cover a DIALOG database or group of databases and would be taught by the 
DIALOG Customer Services staff (as opposed to a person from the database producer). 
If so, please indicate your preference on these topical sessions below. 

_ Patents _ Life Science _ Engineering/Physics 

_ Environment _ Food/Agriculture _ Education/Social Science 

_ Other: ___ 

7. Any other comments on today's session or the DIALOG service: 


4 


Thank you for your time. Please return this form to the DIALOG instructor. 

Please continue comments on the reverse side. 








(DEFENSE TECHNICAL INFORMATION CENTER) 
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DEFENSE DOCUMENTATION CENTER 
DEFENSE RDT&E ON LINE SYSTEM TRAINING EVALUATION 

You have just completed a training program to learn to operate the Defense RDT&E On-Line system. DDC is interested 
in your constructive comments concerning course content, format and presentation so that future instruction can meet your 
needs. We would appreciate your frankness and cooperation in completing this evaluation. 


A. TRAINING RECEIVED 

1. TYPE OF TRAINING 

□ RETRIEVAL 
3. DAT A BASE(S) 

□ "R FILE (1473) 

□ WU FILE (1498) 


PRELIMINARY 


□ INPUT 

O PP FILE (1634) 

□ IR&D (271) 


2. LEVEL OF TRAINING 

□ BASIC CD REFRESHER CD INTERMEDIATE CD ADVANCED 


4. LENGTH OF COURSE 
CD 3 DAYS 
CD 5 DAYS 


CD 10 DAYS 
CD 2 WEEK! 


B. YOUR CURRENT POSITION 

1. ORGANIZATION 


CD GOVERNMENT CD CONTRACTOR 


CD OTHER 


2. TITLE OF YOUR POSITION 
CD LIBRARIAN 
CD RESEARCH SPECIALIST 


CD TECH. INFO. SPECIALIST 
CD SCIENTIST 


C. REMOTE TERMINAL INFORMATION (Preienl or on order) 

1. EQUIPMENT ' 


□ U-100 

□ U-200 

3. OTHER ON LINE SYSTEMS 


O COP PRINTER 
□ 800 PRINTER 


CD CLERK-STENO 
CD PROGRAMEP 


CD ADMINISTRATIVE 
CD OTHER 


□ NASA/RECON 


□ DIALOG 


n SOC 


2. LOCATION OF EQUIPMENT 

□ LIBRARY □ STAFF AREA 

□ COMPUTER ROOM □ OTHER 


□ ORBIT 


D OTHER 


4. WHICH ONE(S) CAN YOU OPERATE? 


DID THIS KNOWLEDGE DhELP □ HINDER YOUR LEARNING THE DDC SYSTEM? 


A. TRAINING MATERIALS 

1. REFERENCE GUIDE 


2. OPERATOR'S MANUAL 

3. REFERENCE TOOLS 


4, VISUAL AIDS 

5. TRAINING ROOM 


COMMENTS 


COMMENTS ON TRAINING 

I ADEQUATE 


NEEDS IMPROVEMENT 


B. OPERATION OF EQUIPMENT 

1. U-100/U-200 _ 

2. KEYBOARD _ 

3. PRINTER _ 

4. SOUND PAGE RECORDER 

5. TAPE CASSETTE _ 

COMMENTS 


DIFFICULT 


NEEDS IMPROVEMENT 


C. SOFTWARE FUNCTIONS AND COMMANDS 

COMMENTS 


□ EASY 


D difficult 


D NEEDS improvement 


PREVIOUS EDITIONS ARE OBSOLETE 






D. HANDS-ON PRACTICE 

□ HELPFUL 

Cl NOT NEEDED 

COMMENTS 




E. DATA BASE COVERAGE 


l. tr file <1473) 


2. WUIS (1498) 


3. PP FILE <1634) 


4. IR&D (271 


ADEQUATE 


NEEOS IMPROVEMENT 



G. LENGTH OF COURSE 


1. □ LEAVE AS IS □ LONGER (NO. of days_ 


2, SUGGESTED ADDITIONAL SUBJECTS: 


. ) □ SHORTER (No. of days 


3. WHAT WOULD YOU OMIT? 


H. ADDITIONAL TRAINING NEEDED 


} ‘ ‘ ' ‘ " — ' -- 

1. □ YES 

2. □ REFRESHER 

3. □ IN 3 - 6 MONTHS 

□ NO 

O INTERMEDIATE 

□ IN 6 - 10 MONTHS 

------ 

□ ADVANCEO 


PART III 

EVALUATION OF INSTRUCTOR 



INSTRUCTOR’S NAME 

DATE 

1. ORGANIZATION OF SUBJECT MATTER 

□ AOEQUATE 

□ GOOD 

□ NEEDS IMPROVEMENT 

2. OELIVERY OF SUBJECT MATTER 

□ ADEQUATE 

□ GOOD 

□ NEEOSIMPROVEMENT 

3. ATTITUDE OF INSTRUCTOR 

□ FRIENDLY 

□ HELPFUL 

□ DISTANT 

4. ATMOSPHERE 

□ FORMAL 

□ INFORMAL 


5. PRACTICE PROBLEMS 

□ HELPFUL 

□ ADEQUATE 

□ NOT NECESSARY 

COMMENTS 

PART IV 

EVALUATION OF COURSE 


□ AOEQUATE 

□ GOOD 

□ VERY GOOD 

□ EXCELLENT 
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(NATIONAL LIBRARY OF MEDICINE, QUESTIONNAIRE FOR INITIAL TRAINING COURSE) 

ON>L!ME SERVICES TRAINING 
Course evaluation 

The National Library of Medicine would greatly appreciate your consenta, 
suggestions, and criticisms on Che following: 

Course content: 


Sequence of topics: 


Uands-or. exercises and follow-up discussions: 


o 
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(National Library'of Medicine, Initial On-Line Services Training) 


Count evaluation •• 


Hand-out* , written extrelaes, and other loatructional old*: 


Instructors: 


Any additional coonenta: 


Name (optional) _______ 

Cetea of training course 


HI* 1 



(NATIONAL LIBRARY OF MEDICINE) 
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ADVANCED ONLINE TRAINING CLASS EVALUATION 


We appreciate your comments on the training you have received at 
NLM and any suggestions you may wish to make for improvements or 
changes in future training courses. 

1. Course prerequisites: 


2. Systems mechanics-overview: 


3. Chemical/Toxicological Searching: 

a. CHEMLINE 

b. TOXLINE 

c. RTECS 

d. TDB 


4. Special application clinics 
MEDLINE/MeSH 

CANCERLINE: 

SERLINE: 

CATLINE, AVLINE, NAF: 


BIOETHICS, HISTLINE, HEALTH, EPILEPSYLINE: 



- LOJ - 

(National Library of Medicine, Advanced Online Training Class Evaluation) 

Page 2 

6. Search formulation sessions: 


7. Topics which should be added or omitted: 


E. Aspects of the course you particularly liked or disliked: 


9. Any additional comments or suggestions: 


Your name (optional) and dates of training: 
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(University of Dayton Research Institute) NAME 

1979 to Pxd6e.nt (August 1979 on) 
BASIC CLASS EVALUATION 


Please mark the one response which most accurately describes 
what your major involvement with the CIRC II system will be. 
(Check one) 

rn ) 

a. 

I will search CIRC II to obtain my own information 
which I will use for other tasks. 

[19] 

b. 

I will search CIRC II to obtain information for 
someone else. 

[ n 

c. 

I will use information from CIRC II which someone 
else obtains for me; I will not search the system 
myself. 

[20) 

d. 

I must know something about CIRC II to do my 
job but I do not regularly search the system or• 
use the information it contains. 

[ 5] 

e. 

Other (please describe). 


2. I anticipate my average weekly usage of the system (for both 
secure and non-secure sessions) will be: (Check one) 


I 5] 

a. 

0 hours 

r J°] 

b. 

less than a half hour 

[ 14] 

c. 

1/2 - 2 hours 

[ 9] 

d. 

2-5 hours 

r n 

e. 

5 hours or more. 


1 
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Please rate your level cf understanding cf the 
following features: 

(UDRl, Basic Class Evaluation) 


1. sign-on 

2. ..off 

3. ..off cent 

4. . ..search 

5. paragraph searching 
€. ..select 

T. cr-er.c-not 
S. par-sen-aej 
9. rec*refereneinc 

10. parentheses 

11. -.site nixed operators with 

bacXreferencing t 
parentheses 

12. ..trovse 
12. newspec 

14. ..print 

15. ..nail 

IS. truncation 
1“. roct 
IS. literals 
IS. ..display 

20. ..puree 

21. ..chance 

22. ..save/..exec 

23. ..cement 

24. ..send/..reev 

25. ..help 

26. C5A0 
21 . fACl 

28. cssr 

29. XLAX 

30. ?TtN paragraph 

31. ?FCN equates/relates 

32. ?*tN relationship and 

attribute codes 

33. using the ?FDN codes 

(searching with) 

34. using the ?F1N aicrofiche 


Bad Scae 
Difficulty at 

Still Somewhat First, but Z Now 
Confusing Understand 



11 

11 

1L 

11 

26. 

II 

m 

11 

77 


Ji. 

2 

_2 

_ 2 . 

_3 

"~7 

_2_ 

_ 2 _ 

_2_ 

_ 4 _ 

_ 9 _ 

_1_ 

_± 

_£ 

17 _ 

11 

11 

21 


J_5 

_22 

11 

11 

1L 

11 

11 

11 

21 

ll 

It 

11 

11 

It 

11 

21 

II 

11 

11 

11 

IS 


22 

11 

15 


IS 

, IS 
19 


I Understood 
From the Start 


II 

36 

JUL 

II 

11 

11 

21 

23 . 

II 

21 


11 

11 

11 

11 

21 

11 

11 

21 

II 
1L 
11 
11 
It 
21 
ZL 
21 

III 
_6 
_6_ 
_5_ 

-1 

_L 

11 

-I 


2 








n ; .1 
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4. Ter these features in Question #2 which you narked "Still 

Scnewhat Confusing", please list their number(s) in the space 
next to the most accurate response: 

3 f 7 >; 28(5); 29(6) Wasn’t discussed in class. 

22,23,26,27,33(2); 

: 17 lil _, Was mentioned, but needed greater explanation. 

22,23(2}; 5,7,8,9,11, . ^ , 

3f. 3? 33M1; 31( 5) _ I tried to use it m terminal practice 

problems, but it didn't work well. 


re tnere 
..eve been 


any commands or techniques which ycu feel 
covered in greater depth -in class? 


should 


3 






(UDRI Basic Class Evaluation) 

-hich aspects covered in the course will be particularly 
helpful once you are back on the job? Were they presented 
clearly anc completely enough that you will feel at ease 
using them? 


a. 


Were any of the materials distributed in class particularly 
helpful or useless? Which ones? 


b 


Should any additional information be covered in handouts 
that at present is not? 




(UDR1 Basic Class Evaluation) 


' as 

ate.ness of ten.;iral practice problems (check as 
applicable). 

y. 

_ a. 

well placed. 

7 

w 

w • 

did not come at appropriate times during 



the course. 

7 

_ c. 

too simple. 

1 

_ d. 

too complex. 

26 

_ e. 

level of difficulty was about right. 

0 

■F 

• 

too many problems. 

S 

_g- 

net enough problems. 

21 

h. 

the problems were fine. 


ZNTS 


vrhat, ir. your opinion, makes a good searcher? How did the 
training package help you develop those characteristics? 

In what wavs can it be improved? 



(University of Dayton Research Institute) 
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NAME 


7 979 to Pfititnt (November 1979 on) 
ONLINE PROFILE EVALUATION 

l. Please rate your level of understanding of the following 


STILL SOMEWHAT 
CONFUSING 


HAD DIFFICULTY AT 
FIRST BUT I NOW 
UNDERSTAND 


rsr'v 

r • \\j. i 


1. customer 
identification 
information 

2. initial profile 
construction 

3. ..END 

4. ..SAVE procedures 


10 


11 


■I FI CATION 


o. 


c . 
7. 
S. 
9. 
10 . 
11 . 
12 . 


function of PCN 
(profile control 
number) 

..EDIT 

MODIFY subcommand 
ADD 

DELETE 
REPLACE 
LIST . 

INDEX subcommand 
..OFF TEMP 


12 


10 


11 


8 

IT 


PROL 




16. 


17. 


use of logical 

operators 

use of truncation 

document structure 

a. PRN (Profile 
Reference 
Number) 

b. Comment 

c. PROFILE 
COPY function in 
..EDIT 


11 


10 


11 


features: 


UNDERSTOOD 
FROM THE START 


21 


17 

11 

18 


11 

1L 

H 

1± 

11 

11 

11 

11 

20 


22 . 

21 

U 


11 

11 

15 




- no - 

(UDRI, Online Profile Evaluation) 

For Those features in Question 1 which you marked "Still somewhat 
confusing", please list their number(s) in the space next to the 
most accurate response: 


_Was not discussed in class 

2(7); 7-1211); 74(7) N eeded greater explanation 

7(71; 79(1) _I tried to use it in terminal practice, but am 

still confused. 


Are there any commands or techniques which you feel should have been 
covered in greater depth in class? 






(UDR1, Online Profile Evaluation) 


- Ill - 


Were any of 
Which ones, 


i 


the materials distributed in class particularly helpful, 
and why? 


Were any materials particularly confusino? Which ones, and why. 


Please include here any additional conments you may have in regards to 
materials covered. 


3 






(University of Pittsburgh On-Line Training Center) 

Evaluation Form 


We would like to have your personal opinion on various aspects of 
the training session. 


1. Presentation of material was: 


Too fast 


About right Too slow 


2. Amount and frequency of review: 


Overdone 


About right Inadequate 


3. Did you feel your time on the Emulator was: 


Too long 


Enough 


Not enough 


Too much 
supervision 


Right amount Not enough 

of supervision supervision 


4. Concerning your time on the Commercial system: 


Too long 


Enough 


Not enough 


Too much 
supervision 


Right amount Not enough 

of supervision supervision 


5. Concerning the total duration of the training period: 


Too long 


About right 


Too short 




- 113 - 
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6. I feel that I can do an adequate search for a user on the 
commercial system. 


Yes 


No 


Perhaps 


7. As a result of this training session I now feel that with the aid 
of a printed manual, 1 could teach myself another commercial system: 


Yes No Perhaps 

8. We would appreciate any comments or suggestions for the improvement 
of the training session. 




American Society for Information Science 
EVALUATION FORM FOR EDUCATION ACTIVITIES 
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COURSE TITLE. 


A. PERSONAL INFORMATION 

1) Current employment status: 
Description of position_ 


2 ) 


Years of experience as an information professional_ 

How would you assess your knowledge of the material covered in this course? 
(check as many as apply) 


Before 

After 


Course 

Course 


□ 

□ 

Unfamiliar with the material. 

□ 

□ 

Familiar with the material in a very general sense. 

□ 

□ 

Familiar with the material and the basic concepts/theories and/or methods/techniques 
underlying it. 

□ 

□ 

Understand many of the concepts/theories and/or methods/techinques underlying the 
subject of this course and able to apply them in practice. 


.substandard 


B. GENERAL ARRANGEMENTS 

1) How would you rate the pre-course information and orientation? (Circle only one number) 

1 2 3 4 5 

-1-_j-1-J-»- 

Poor Average Excellent 

2) How would you rate the facilities (room, living accommodations, food) provided for the course? 

_outstanding _comfortable _adequate 

_substandard _not applicable 

3) How would you rate the audiovisuals, handouts, and other materials? 

_outstanding _adequate 

C. COURSE ARRANGEMENTS 

1) Was the course what you expected from the announcements? 

_yes _too elementary 

2) Was the level of the course suitable for your needs? 

_about right _too elementary 

3) What is your opinion of the total work load of the course? 

_about right _too heavy 

4) What is your opinion of the length of time allowed for the course? 

__about right _too long 

5) Rate the balance of time allowed for each topic. (Circle only one number) 

1 2 3 4 5 


_too advanced 


too advanced 


.too light 


too short 


Poor 


Average 


6) Were the presentations scheduled in logical sequence? (Circle only one number) 
12 3 4 


Excellent 


Poor Average Excellent 

7) Was the variety of presentations sufficient to maintain your interest? (Circle only one number) 
1 2 3 4 5 


Poor 


Average 


Excellent 




8) Was the course practical in terms of your needs? (Circle only one number) _ ^ 3 _ 

1 2 3 4 5 

_j_1_1-1-1— 

Poor Average Excellent 

9) Did the course provide adequate theory on the subject? (Circle only one number) 

1 2 3 4 5 

_ 1 _ 1 -1_1_1_ 

Poor Average Excellent 

D. SPEAKERS (Rate the speakers on the following aspects of their presentation on a scale of 1 (''poor”) to 5 (“excellent") 
with 3 being "good") 

Command of Clarity and 

Speaker Preparation Subject Organization 



E. ACHIEVEMENT OF COURSE OBJECTIVES (Refer to the stated objectives of the course and rate their 
degree of achievement by circling the number on the scale that reflects your opinion) 


Not achieved 


F. SUMMARY EVALUATION 


Partially achieved 


Fully achieved 


1. How would you rate your satisfaction with the course as a whole? 

1 2 3 4 5 

— 1_1_1_1_j 

Dissatisfied Satisfied Very satisfied 

2. General comments (Add below any further comments you may have on any aspect of the course) 


G. FUTURE COURSES 

Are there any other topics or subjects that you would like to see presented in the ASIS Continuing Education Program? 
If so. please state below. 


loptional) 

PLEASE RETURN TO: ASIS CONTINUING EDUCATION PROGRAM, 1010 Sixteenth St.. N.W.. Washington, 
DC 20036 


J 



(UNITED STATES GOVERNMENT) 


11' 



REQUEST, AUTHORIZATION, AGREEMENT 
AND CERTIFICATION OF TRAINING 


L Agency, corn agency subelemen! •) I 1 I 
ond submitting office number l——| 

(uunv/t-u-u-wil I 



1. Apphcant's nam (Last Fmt Middle Ininal) 

MKAte preferred title Mn. Mr. Mi etc) 


4. Home address (NMmbrr. Street, etty. State, ZIP code) 


' ^ OtgenuetWi mawflf address (Branch non/Office/Bureau/Agency ) 


IV Position title/tunction 


Section A—TRAINEE INFORMATION 


Inter first It) l Social Security Humber 

5 jotters ol l—] 

fait name 


), Home t el ephon e 


Area code ; Humber 


I — J Date ot 


birth ( Year and month j 


(example—bom 
January 14. 194) 
shown as 4J/0I 


%. Position level (Mark (X) oat onlyl 


a. Hon supervisory J I c. Manager 


4. Ixecutr* 


II. Humber of pnot. 
nongovernment 
t»e*rwAf bars 



Section B—TRAINING COURSE DATA 


lie. Hame and mailing address of training vendor (So., ureei ary. Stare. ZIP code) ; 111. Location of training site (If same mark taxi 


M. Course title end training objectives (Benefits to be deneed by the Government} 


i 17. Celato«/Course Ho 


It- framing period (6 digits) 


II. Ho of course hours (4 digits) 








•7 | H Trammg codes (See instructions) 



AREAS OF EVALUATION 

(Place (<0 i* dffeofnate column to indicate your evaluation of Hems 26 through ) 7 Do not attempt to spin a rating) 


M Stated ebtectme 


A Yes 


Excel lent 


B — Partial ly 


B = Sufficient 



M Ortmtbtiob el 

eubtact matter a - Well organiTed B — Adequate c - Poorly organized 





M SurtaMify of 

metructionei materials A ticeffent B = Adequate ( - Poor 


1 

■ 

i 

! 


Ml Lever o» drtficwty 


11 Lmtgth of course 


Too advanced 

B * Appropriate 

C - Too Ham antary 

Too long 

B r Appropriate 

C - Too short 


M AdpHcaMrfy af ml 
matter to tfce rob 


M faoWre* 


A Significant 


- EeceHent 


Mighty recommend 


* Yes 


B - Adequate 


B — Good 


B Recommend 


8 No 


C - mmtnificant 


C - Hot re co mmended 


t • Hof appHcjbtf 



M Amount of outside 
ef owning mere 

A Too much 

B 

- Appropriate 

C • Insufficient | 

! 

! 

■ 

. 

M effectiveness of 
metrecton 

A ticeiient 

a 

Good 

C - Poor I 

- 1 

-~i 

!— l — j 

- 1 -. 

i 


Copy •—AOCNCY (EVALUATION) 


1 no Pints' 

U S C'vii Service Commission FPM Chaplet 410 
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APPENDIX C 

ADVANTAGES AND DISADVANTAGES OF 
INSTRUCTION METHODOLOGIES 

The pages that follow reproduce material from the report, 
Man ageri al and Supervisory Training Techniques and Methods 
U.S. Civil Service Commission, Bureau of Training, 1977, 
on the following methodologies: 

- Lecture 

- Slide/Tape Presentation 

- Instructional Film 

- Instructional Television 

- Self-Paced Instruction 

- Coaching 

- Planned Exercises 

- Correspondence Study 
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Lecture • Lecturette 

The lecture, an oral presentation prepared and delivered by a subject-matter expert, is probably 
the oldest and most basic form of instruction. It is used to supply the greatest amount of information in the 
least time. Like other types of information presentation, it does not allow for feedback from the learners; 
thus, it should be combined with participative techniques whenever possible. 

How It Can Be Used: 1. To introduce concepts, identify and analyze problems, or clarify issues 

related to supervisory and managerial functions. 

2. To deliver training content which is best presented all at once in an orderly 
manner and does not require practice. 

Who Can Benefit from 

It: 1. Trainees who have similar needs and similar capabilities to learn the 

material. 

2. Managers and supervisors who are.highly motivated to learn but short of 
time. 

1. Lectures are efficient in terms of time, facilities needed, and the number of 
participants who can be trained at one time. 

2. The lecturer retains control over the learning content. 

3. This technique may be more acceptable than others are to participants who 
are accustomed to a traditional teaching-learning situation. 

4. Lectures can easily be varied to suit trainees' needs and can be used with 
almost any other training technique. 

1. Participants usually have no opportunity to make comments, ask questions, 
or otherwise show that they understand the material presented. 

2. The lecturer must be qualified both as a subject-matter expert and as a 
speaker. 

3. Long lectures (over 40 minutes) may not be readily learned or remembered. 
Lecturettes are shorter and can be highly effective. 

4. Listeners might not make the transfer from intellectual understanding to 
practical application. 

1. Some books dealing with supervisory and managerial training contain 
outlines for lecturing on pertinent topics. 

2. Lectures are often more effective if supplemented by visual materials such 
as handouts, slides, chalkboard notes, etc. 

1. Analyze audience needs. 

2. Determine objectives. 

3. Plan lecture. 

4. Present lecture, listing main points and then expounding on them. 

5. Summarize main points. 

For More Information: 

International Labour Office. An Introductory Course in Teaching and Training Methods for Management 
Development. Geneva, 1972. 

Pfeiffer, J. William, and John E. Jones. The 1972, 1973, 1974, 1975, 1976 Annual Handbook for Croup 
Facilitators. La Jolla, CA: University Associates Publishers, Inc., 1974 (Also 1972-76). 

Staton, Thomas R, How to Instruct Successfully: Modem Teaching Methods in Adult education. New 
York: McGraw-Hill. 1960. 


Advantages: 


Disadvantages: 


Materials: 


Procedures: 


21 
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The Slide/Tape Presentation 

With the advent of inexpensive, portable machines for synchronizing still visuals and sound, the 
most basic audiovisual system was created. Although relatively new, the slide/tape presentation is now in 
wide use for both group and self-paced instruction. It is used in conference rooms, in classrooms, in 
individualized learning centers, and for on-the-job training. Several slide projectors can be used with a 
controller and tape player for a multi-screen presentation. 

How It Can Be Used: 1. To reinforce information presented orally with visual cues. 

2. To stimulate interest in the topic being discussed. 

Who Can Benefit from 

It: All levels of managers and supervisors. 

Advantages: 1. Slides and tapes can be presented together or separately; for example, the 

slides can be used first with the tape and later, as the narrative changes, with 
a lecture. 

2. After the initial equipment and taping expenses are met, the presentation 
can be repeated many times at little cost. 

3. The slide/tape production equipment is relatively inexpensive. 

1. The slide/tape show by itself does not allow for discussion, interaction, or 
practice unless it is specifically designed to do so. 

2. Two pieces of equipment must be used together, possibly causing problems 
in coordination. 

3. Preparation of tapes, slides, and scripts requires considerable expertise; most 
training offices have to contract for the service from an outside source. 

1. A slide-synchronizing tape player plays a tape with prerecorded advance 
pulses that control a compatible slide projector. With non-automatic 
equipment slides are advanced manually. 

2. Slide/tape presentations for supervisor and manager training are available 
from firms which produce instructional media. 

1. When used with groups, the presentation may be introduced, allowed to run 
without interruption, and then discussed. If the presentation is a long one. 
the trainer may wish to stop at intervals for a question and answer period. 

2. The slide/tape presentation may also be used with workbooks in self-paced 
instruction. 

Related Approaches: Slide/tape presentations (sometimes called Slide/Sound Shows) are often used in 

Programmed Instruction. Also, see references for Audiovisual Aids and 
Techniques. 

For More Information: 

Bergevin, P., D. Morris, and R. M. Smith. Adult Education Procedures: A Handbook of Tested Patterns for 
Effective Participation. New York: The Seabury Press, 1963. 

Otto, Calvin P., and Rollin 0. Glaser. The Management of Training: A Handbook for Training and 
Development Personnel. Reading, MA: Addison-Wesley, 1970. 

Pula, Fred John. Application and Operation of Audiovisual Equipment in Education. New York: John 
Wiley 8t Sons, Inc., 1968. 

Rigg, Robinson P. Audiovisual Aids and Techniques in Managerial and Supervisory Training London: 
Hamish Hamilton, Ltd., 1969. 


Disadvantages: 


Materials: 


Procedures: 


22 





The Instructional Film 


The practice of presenting information through films has been widely accepted by the training 
community for many years. Films offer the advantages of virtually all other media in an easy-to-use format. 
Films can combine the benefits of other means of presentation (still visuals, print, lecture, etc.) to enhance 
instructional content. 


How It Can Be Used: 


Who Can Benefit from 
It: 

Advantages: 


Disadvantages: 


Materials: 

Procedures: 


Related Approaches: 


1. To demonstrate ideal ways of doing such tasks as interviewing and 
counseling. 

2. To illustrate new concepts and techniques. 

3. To take the place of the instructor when he or she cannot be present. 

4. To teach human relations, an important part of supervisor and manager 
training. 

Films can be used in virtually any training situation. 

1. Films standardize the dissemination of information; variations among 
instructors are eliminated. 

2. The person who runs the film can control time and space in ways 
unparalleled with other media—animation, magnification, slow and fast 
motion, etc. 

3. Of all information presentation techniques, films are the most likely to 
catch and hold the full interest of the participants. 

4. A lecture, demonstration, skit, or discussion that is filmed can be used many 
times. 

1. Since the training room usually must be darkened during the film, the 
instructor misses the trainees' immediate reactions to it. Also, trainees may 
not be able to take notes. 

2. Equipment and facilities for presenting films are not always available. 

3. Unless trainers are able to contract for the production of special films, they 
must rely on standard training films in which some of the content may be 
inappropriate. Sometimes, but not always, the trainer may be able to use 
just part of a film. 

Films are available from a number of instructional media firms. Most 

Government agencies with training functions have film catalogs. 

1. Preview films to be sure that they contribute to the fulfillment of the 
learning objectives. 

2. Introduce the film and specify learning objectives. 

3. Allow time for the participants to discuss the film after viewing it. 

Movies, Training Films, and Audiovisual Aids. 


For More Information: 

Bergevin, P., D. Morris, and R. M. Smith. Adult Education Procedures: A Handbook of Tested Patterns for 
Effective Participation. New York: The Seabury Press, 1963. 

Brown, James W„ Richard B. Lewis, and Fred F. Harcleroad. AV Instruction: Media and Methods. New 
York: McGraw-Hill, 1973. 

Otto, Calvin P., and Rollin 0. Glaser. The Management of Training: A Handbook for Training and 
Development Personnel. Reading, MA: Addison-Wesley, 1970. 

Pula, Fred John. Application and Operation of Audiovisual Equipment in Education. New York: John 
Wiley & Sons, Inc., 1968. 

Rigg, Robinson P. Audiovisual Aids and Techniques in Managerial and Supervisory Training. London: 
Hamish Hamilton, Ltd., 1969. 






Instructional Television 


Instructional television is comparable to film in its uses and advantages. An effective television 

presentation brings the action to the viewer; trainees have a chance to observe things which would be very 

difficult to replicate in a classroom. Like other means of audiovisual presentation, television can be used in 

individual instruction, particularly when the tapes are in video cassette form. 

How It Can Be Used: 1. To record a role play, discussion, or presentation and replay it for detailed 

examination. 

2. To record a session for viewing by persons unable to attend. 

3. To demonstrate desirable performance in a more realistic setting than the 
classroom. 

4. To take the place of the instructor when he/she cannot be present. 

Who Can Benefit from 

It: Supervisors and managers at all levels. 

Advantages: 1. Television can take management training out of the academic realm and 

place it in an environment which more closely resembles the real world. 

2. Presentations can be taped ahead of time; trainees can replay parts of the 
tape as often as needed. 

3. In the form of video cassettes, television presentations are more convenient 
and often less costly to produce than films. 

4. Video cassettes are more adaptable to small groups than is film. 

5. Video cassettes can be set up and operated more easily than film can. 

6. Most trainers can handle the mechanics of showing a television presentation; 
thus it is easy for them to tailor a presentation to the needs of a particular 
group. 

Disadvantages: 1. The videotape player and monitor are more, ex pensive than a film or a slide 

projector and screen. 

2. The viewing area is limited to the size of the monitor. It may be too small to 
be seen by large groups. 

Materials: 1, Video cassette tapes for individual instruction are available from training 

media sources. 

Procedures: 1. Preview prerecorded tapes as you would a film. 

2. Introduce the tape and go over the learning objectives. 

Related Approaches: Closed Circuit TV, Audio Cassette Tapes, Programmed Instruction, Audiovisual 

Aids. 

For More Information: 

Brown, James W., Richard B. Lewis, and Fred F. Harcleroad. AV Instruction: Media and Methods. New 
York: McGraw-Hill, 1973. 

Otto, Calvin P„ and Rollin O. Glaser. The Management of Training: A Handbook for Training and 
Development Personnel. Reading, MA: Addison-Wesley, 1970. 

Pula, Fred John. Application and Operation of Audiovisual Equipment in Education. New York: John 
Wiley & Sons, Inc., 1968. 

Rigg, Robinson P. Audiovisual Aids and Techniques in Managerial and Supervisory Training. London: 
Hamish Hamilton, Ltd., 1969. 
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Self-Paced Instiuction 

Self-paced instruction is a training method in which the content is presented by means of 
programmed study materials rather than by an instructor. It can be used by an individual or integrated with 
group instruction. Programs have the following features: 

1. The material to be presented is carefully and logically organized. 

2. The content of the program is presented in segments or exercises. Before the learner can 
proceed to the next segment, he/she must respond to the first one. 

3. Learners find out immediately whether they have mastered each segment of material. 

Self-paced instruction uses a variety of media, including books, audio tape, slides, films and television. 
Quite often, combinations of media are utilized. Programs are prepared according to the following 
principles: 

1. The program is designed to accomplish a set of objectives. These objectives are precise 
statements of the behavior to be learned by the student. Objectives are derived from an 
analysis of the tasks to be performed on the job. 

2. The program is tested. If the learners do not score well on criterion test items or exercises, the 
program is revised and retested. This process continues until a sampling of the target 
population is able to complete the program at a predetermined level of achievement. 

How It Can Be Used: 1. To teach principles, concepts, and techniques that do not require practice or 

demonstration. 

2. To give learners the information they may need as background if they are to 
profit from such other techniques as group discussion, role playing, and case 
studies. 

With appropriately developed programs, self-paced instruction can be used for all 
levels of supervisors and for mid-level managers. 

1. This method can be adapted to the individual's needs and to the amount of 
time that is available for training. 

2. Because so many media and sources of information can be used, the training 
content for self-paced instruction is almost unlimited. 

3. Getting immediate feedback on what has been learned increases the learner's 
confidence and motivation. 

1. Self-paced learning should be supplemented by other training approaches if 
practice in applying the learning is needed. 

2. Whether the training is used on the job depends on the motivation of the 
individual learner. 

3. The cost of developing self-paced instructional materials can be high. 

4. Developing self-paced instruction involves analysis and specification of 
behavioral objectives. It also involves designing, testing, and retesting 
materials. This requires more time, effort, and talent than many other 
methods of instruction. 

Procedures: 1. Before the learner begins studying the material, he/she is introduced to both 

the material and any equipment that he/she may have to use. If the course is 
taught in a learning center (LC), the LC coordinator responds to 
student-initiated requests for information and/or clarification, observes 
progress, and provides necessary assistance. If subject matter experts are 
available, the coordinator will arrange for them to be available to the learner 
if and when needed. 

2. Self-paced instruction can be integrated with conventional group methods: 
after completion of a segment of materials the instructor conducts group 
discussion designed to reinforce participant learning gained through use of 
the programmed material. 

Programmed Instruction, Personalized Self-Instruction, Programmed Learning. 


Who Can Benefit from 
It: 

Advantages: 

Disadvantages: 


Related Approeches: 
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Coaching 

Coaching usually involves a one-to-one relationship between the “coach" (a supervisor or manager 
who is more experienced or at a higher level than the person being coached) and the understudy, but 
coaching sessions can be held for groups as well. Besides the obvious benefits to the understudy's 
professional development, the time spent working closely with a superior often makes the participant more 
confident about his or her own abilities and more comfortable about dealing with other superiors. Coaching 
is an ideal form of training for supervisors and managers in that they gain experience in dealing with 
situations as they arise, yet there is constant supervision from a knowledgeable coach. 

How It Can Be Used: 1. To promote the interchange of information between different levels of 

supervision or management. 

2. To prepare an understudy to assume a particular position. 

3. To continue particular managerial philosophies or approaches. 

Who Can Benefit from 

It: 1. Newly appointed supervisors and managers. 

2. Persons chosen to assume the same position as the coach's but in another 
location. 

Advantages: 1. The coaching can be geared specifically to an individual's needs and 

capacities. 

2. Contact with the coach is nearly continuous. 

3. The learner has the benefit of both his/her own experience and that of the 
coach. 

Disadvantages: 1. How valuable the training is depends to a large extent on the coach's 

competence as a coach. - - 

2. It is sometimes difficult to get supervisors or managers at higher leve's to 
devote the large amount of time needed for good coaching. 

3. In some instances, the understudy is merely an observer or mimic of the 
coach's behavior. 

Procedures: t. The coach and the understudy agree on what will be required of each during 

the coaching. 

2. The understudy's learning needs are prioritized and the coach is advised of 
the priorities. 

3. If desired, a written training plan can be worked out. 

4. The training office assists by planning learning objectives, by supplying 
developmental materials such as reading lists, and by conducting special 
developmental activities. 

Related Approaches: Understudy. 

For More Information: 

Craig, Robert L. (Ed.). Training and Development Handbook (Sponsored by ASTD) New York 

McGraw-Hill, 1976. 

Kirkpatrick, Donald L. A Practical Guide for Supervisory Training and Development Reading, MA 

Addison-Wesley, 1971. 

Singer, Edwin J. Effective Management Coaching. London: Institute of Personnel Management, 1974. 
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Planned Exercises 
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Planned developmental experiences offer the advantages that job rotation and coaching offer, but 
most of them do not require that the employee leave his or her present position or have the continuous 
attentior of a superior, A well designed program of planned experiences can have the same results as 
coaching or job rotation. Some types of planned experiences are: 

Special assignments or special projects (for example, a study to determine the advisability of setting up a 
new organizational unit). These assignments can be used equally well for developing employees' skills in 
specific areas of interest or for familiarizing them with the overall operations of the organization. 

Work on committees or task forces. This provides experience in working with others and sometimes in 
researching and preparing reports. Often employees make important contacts with others in the field 
through such work. 

Membership in professional societies, advisory groups, etc. While representing his or her organization at 
meetings, the participant is promoting professional contacts and is keeping up on developments in the 
profession. 

Assignments as acting manager or supervisor. This is an excellent way to fill the position of a temporarily 
.ib'ent supervisor or manager while training the subordinate at the same time. The subordinate should be 
-hi full responsibility when possible. 


How They Can Be Used: 1. To provide experience that the employee cannot get in his or her current 

position. 

2. To enhance the participant’s abilities in the areas of problem solving, 
cooperation, and advocacy. 


Who Can Benefit from 
Them: 


Advantages: 


Disadvantages: 


Procedures: 


1. Depending on the experiences provided, all levels of supervision or 
management can profit from them. 

2. This approach is especially beneficial for persons whose present jobs are 
somewhat narrow. 

1. Assignments that allow for new experiences help supervisors and managers 
to assess their own performance in different situations; they also can 
provide indications of competencies in other areas. 

2. Trainees usually gain from the outside experiences skills that help them in 
their regular jobs. 

3. A person's development will be more rounded if he/she has had a variety of 
planned experiences. 

1. Whether an individual’s developmental objectives can be met depends on the 
availability of appropriate experiences. 

2. Special assignments must be made carefully so as not to infringe on the 
responsibilities of other organizational units. 

3. An assignment may "get the job done" but not yiold a planned experience. 

1. The training office can suggest possible experiences to the supervisor of 
persons for whom the experiences would be appropriate. 

2. Information on past assignments and the participant's reactions to them 
should be kept by the training office. 


Related Approaches: Planned experiences are also referred to at Planned Individual Development and 

Outside Assignments. 

For More Information: 

Byers, Kenneth T. (Ed.). Employee Training and Development in the Public Sector. Chicago: International 
Personnel Management Association, 1974. 

Tracey, William R. Designing Training and Development Systems. New York: American Management 
Association, Inc., 1971. 




Correspondence Study 
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In this type of training, the supervisor/manager usually receives the necessary study materials 

through the mail. Working at his or her own pace, the student completes lessons one at a time and mails the 

responses to test items to a correspondence center. The responses are quickly graded (often by machine) 

and returned. Several lessons make up a course or learning module. 

How It Can Be Used: 1. To teach cognitive skills. 

2. To introduce basic information which can be expounded upon later in 
face-to-face instruction. 

Who Can Benefit from 

It: 1. Entry-level management and supervisor trainees. 

2. Managers Shd supervisors who want to familiarize themselves with new 
concepts or techniques. 

Advantages: 1. This approach provides for the regular evaluation of individual learning 

achievement throughout the course. 

2. The employee's work is not necessarily interrupted since studying is not 
usually done during work hours. 

3. Because there are no additional expenses beyond those of the tuition and 
the materials, correspondence study is often less expensive than other 
training. 

4. If employees cannot conveniently be assembled, correspondence courses can 
be used to provide the same lessons to each of them. 

Disadvantages: 1. Prepared correspondence courses which address the particular needs of 

managers and supervisors may not be available. 

2. Instructional objectives for this method are limited to those for which 
competency can be demonstrated through responses to a paper-and-pencil 
test. 

3. Correspondence courses must be supplemented by other training approaches 
when the training required involves developing interaction skills or similar 
participatory skills. 

4. It is often difficult to prepare "student-motivating" instructional com¬ 
ponents. 

5. Completion rates are significantly lower than those for other types of 
training. 

Materials: 1. Emphasis should be placed on selecting courses with the appropriate 

objectives since the course usually cannot be modified to account for 
changes in subject matter or for the needs of individual learners. 

2. Course materials and strategy should be validated for the I earner's 
characteristics and learning environment. This is especially important since 
the student works alone. 

Procedures: 1. The person who arranges for the course should verify that all of the 

instructional materials are available to the learner; some correspondence 
courses require reference material and audiovisual equipment not supplied 
with the lessons. 

2. The student must be told the objectives, the criteria for success, and the 
consequences of late submissions, wrong answers, etc. 

3, If possible, the student should have an opportunity to contact an instructor 
(perhaps by telephone) to ask questions, get feedback on performance, etc. 

Related Approaches: Extension Courses, Home Study. 

For More Information: 

Craig, Robert L. (Ed.). Training and Development Handbook (Sponsored by ASTD). New York: 

McGraw-Hill, 1976. 

Erdos, Renee F. Teaching by Correspondence (A UNESCO Source Book). London: Longmans, Green & Co. 

Limited, 1967. 
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APPENDIX D 

ANALYSIS OF QUESTIONNAIRE RESPONSES 

Introduction 

As is discussed in the Study Methodology section of this report, a 
questionnaire was developed to obtain trainee impressions of the courses they 
attended. The 16 questionnaires (13% return) received were a disappointingly 
small response. They provide too little data to be definitive. Discussion 
of the responses was, therefore, excluded from the main report and appears 
in this Appendix. 

The Questionnaire 

A copy of the 3-page questionnaire is given on pages 91-93 • The original 
was on legal-size paper. 

From a question design standpoint, DTIC may wish to consider the rating 
scale format used for questions 2 and 9 in an evaluation instrument. Provided 
that the number of options is not too large to obliterate distinctions, people 
usually find this format more adaptable to the assessments they want to express 
than a more restrictive format. For example, Part III of DDC Form 345 might 
be amenable to restructuring. 

The Responses 

Tables 1 and 2 summarize the responses to Questions 1 and 14. Not 
unexpectedly, usage of the Lockheed, SDC, and BRS systems correlates with 
training courses that respondents attended. The advanced courses appear 
almost as popular as the introductory courses. Question 14 responses indicate 
a popularity of the New York Times database. The distribution by year of the 
training experiences indicates that perhaps half of the respondents have 3 
and 4 years of searching experience, and almost all appear to have been 
searching for at least one year. Responses to Question 14 showed moderately 
heavy multiple database usage. Five respondents use 5 or more of the online 
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systems, and 4 use 3 of the systems. 

It is difficult yet to know whether the usage reported in Questions 13 
and 14 represent a low potential utilization of online database or a level 
that accords with end-user need. Current weekly usage of online systems is 
shown in Table 3. 

The prevalent system/database use of 0-2 hours/day reported in Question 
14 suggests one group of searchers who average 3/4 to 1 hour of terminal 
activity daily and another group who work at terminals about 3 hours daily. 

A special announcement was made at the Washington, D.C., Online Users 
Group meeting for questionnaire responses in the hope that a good response 
could be obtained from federal librarians. Table 4 shows that this didn't 
happen. Over half of the respondents are from the for-profit sector. 

The most popular, wanted teaching methodologies reported for Question 

3 are: 

Lecture (12 votes) 

Discussion (6) 

Transparencies (6) 

Hands-on practice (4) 

The two most disliked methodologies are: 

Role playing (4) 

Lecture only (3) 

Some who voted positively for lectures specified their preference for lectures 
combined with online demonstrations on CRTs visible to the students. One person 
considered such lectures "ruined" when the computer system went down. Having 
experienced this unsatisfying situation, the author, when teaching, developed 
handout and transparency materials as backups and recommends this precaution 
to DTIC in this report. 

Open-ended questions such as 4, 5 and 6 are a risky way of trying to 
obtain information because people frequently ignore them. Only 2 people did 
not respond in some way to Question 4, The program content mentioned the most 

■I 
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was hands-on practice (6 times) and search strategy examples (4 times). Other 
course content respondents found helpful included: 

- Exchange of experiences with other participants 

- Explanation of how the computer system executes various 

commands 

- Discussion of strategies appropriate for assigned 

problem questions 

- Database design 

- Explanation of databases in the context of applying 

search questions to them 

- Indexing policies (in advanced courses) 

More hands-on practice was the most frequently mentioned suggestion 
(4 times) of program content respondents did not consider adequate. 

Three respondents criticized the "sales pitch" content of courses, one 
suggesting that courses be taught by experienced searchers, not sales persons. 
Others wanted more of the following content: 

- Comparison of various systems (differences, advantages) 

- Discussion of real-life problems, how they occur and 

are solved 

- System documentation 

- Better use of transparencies (removing them too fast 

for note taking criticized) 

Only 7 respondents provided information for open-ended Question 6. 

That this question was passed over might have been predicted because it is 
more difficult. It asks people to reach beyond their experience and into 
their imagination. Content suggestions included: 

- The rationales underlying the structures of databases 

- System and file design (linear and inverted files, 

stopword list, string characteristics, etc.) 

- Subscription price information 

- Hands-on practice 

- Difficult multi-term searches 

- Management issues 

- Historical (this may refer to the development of 

online services or the evolution of particular databases) 





Respondents provided a r nge of 1 to 16 hours as needed to gain a working 
knowledge of a single database, and a range of 2 hours to 4 days for two related 
databases (Question 7). This is a type of question for which computation of 
an arithmetic mean would have little meaning. Of the people who reported 
attending 3 to 8 training courses, 8 individuals specified 8 or more hours for 
a single database. Within this group, 4 people who had attended 6 to 8 courses 
specified a training period of 2 or more days for 2 related databases. It is 
believed that more weight should be given to opinions of those with more 
course experience. 

Table 5 shows the responses to Question 8. Respondents were about 
evenly divided between the sufficiency of one course and a two-course sequence 
provided that updates on system + database combinations are available. This 
response underscores the value found in updates end the importance of making 
them real learning experiences. 

Remarks provided for Question 10 reinforced those for Questions 3, 4 
and 5. Five respondents addressed content rather than type of teaching material, 
mentioning search strategy examples, case studies, prototype searches, and 
search studies such as have appeared in the periodicals, Pa t abase and Online . 
Other responses recommending the following materials: 

- Good manuals (3 mentions) 

- Handouts sufficiently comprehensive to alleviate 

note taking (3 mentions) 

- Transparencies 

- Audiovisual modules 

- Charts 

- Venn diagrams 

- Hands-on quizzes 

- Quick reference pamphlets/cards 

Respondents were negative about self-teaching manuals and CAI replace¬ 
ments for more conventional methodologies. For self-teaching manuals, they 
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TABLE 3 

13. Bow Much online searching do you do? 

Flense circle Che appropriate letter below. 

__L *• °ver 20 hre/wk 4 d. 5-10 hr«/wk 

JbL k. 15-20 hrs/wk I «. 3-5 hrs/wk No response 

c . 10-15 hre/wk 3 f. %-3 hre/wk 

O |. don't do searching 


TABLE 4 


15. What type of organ lest Ion do you work for? 

3 A. For profit (not e DOD contractor) JLf. 
^ b. For profit (DOD contrector) Q g. 
A «. For profit (Information broker) O h. 
oE 4 . Mot for profit (non-ecedemlc) O l. 
3 e. Mot for profit (academic) 


Federal govt, (non-cllltary) 
Federal govt, (military) 
State, local government 
Self employed 


TABLE 5 


8. Which of the following altematlvee do you consider the beet to acquire 
searching proficiency? (Please circle the appropriate letter) 


O 


•, * single couree followed by on-the-job practice 

b. A beginning course, on-the-job practice; 6-12 months later, an advanced course 


C. A beginning course, on-the-job practice; 6-12 months later, an advanced 
course on the command language and system and advanced database courses 
d. Alternative a and periodic 1-day updates on system + databases 


«. Alternative b end periodic 1-day updatas on system * databases 


f. Alternative c and periodic 1-day updates on system + databases 


c 






4 Yes 
6 No 


5 Maybe 
1 No response 


133 


One person’s "maybe" was accompanied by the note "when money is short." To 
CAI, they voted 

2 Yes 2 Maybe 

11 No 1 No response 

It is likely that at least part of the above negativity is caused by a lack of 
familiarity. I had personally thought that CAI would not be an adequate 
substitute for the lecture mode until I used it (on a system previously 
unfamiliar to me). The experience indicates that CAI speeds learning of simple 
facts but it cannot be used for situations where students are likely to ask a 
lot of different questions and need tailored answers to learn. CAI could be 
used to learn basic aspects of a command language and database structure; its 
present drawback is the programming expense. People seem to fear the loss of 
contact with a human tutor. 

Hoped for information about searchers, search behavior, learning needs 
and preferences, and database usage did not materialize from this questionnaire 
survey. The questionnaire, as a questionnaire, seems to have passed a field 
test in that respondents seemed to have little difficulty understanding and 
answering the questions. Write-ins for Questions 1 and 14 point to the need 
to add to the named courses and systems should these questions be asked again. 
DTIC might find is desirable to obtain information about the experiences of 
dialup trainees before they receive DTIC training. A modification of Questions 
l and 14 together with, possibly. Question 4 and Question 13 could provide 
this information. Knowing the experience trainees bring to DTIC’s courses 
might aid the interpretation of responses to DTIC's Course Evaluation ques¬ 
tionnaire (DDC Form 345). 
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programs of the following organizations for examination: (a) Lockheed Information! 
Retrieval Service (LIRS) (2 programs), (b) University of Denver Graduate School 
of Librarianship & Information Management (GSLIM), (c) National Library of Medi¬ 
cine (NLM), (d) University of Dayton Research Institute (UDRI), (e) University off 
Pittsburgh On-Line Training Center (Pitt), (f) House Information Systems (H.I.S.)| 
(g) Congressional Research Service, Library of Congress (CRS), and (h) Defense 
Technical Information Center (DTIC). 

A set of criterion variables was developed by which the instruction programs 
were analyzed and compared. These variables address four major areas: (1) pur¬ 
posive documentation (goals, objectives, criteria), (2) education program (compo-j 
nents, duration, availability, location, participants), (3) resources (human, 
equipment, facilities, instruction materials), and (4) evaluation/feedback mech¬ 
anisms (instruments, tests, follow-ups, user groups). 

Most programs are found to be weak in purposive documentation that limits LheJ 
bases on which their developers can evaluate their accomplishments. Five of the 
programs rely on lectures as the principal mode of knowledge transfer. Pitt and 
H.I.S. programs integrate lecturing with hands-on practice. LIRS's ONTAP and CRS| 
programs provide online self-instruction. Four of the programs have a multi- 
course structure, offering one or more advanced courses beyond the introductory 
one. No course other than the academic one (GSLIM) is longer than one week in 
duration. Only LIRS's ONTAP program is substantially location independent; all 
others require students to travel to either a single location or one of several 
locations. Although six of the courses have no course fees, the high indirect 
costs of travel probably prevent some individuals needing instruction from obtain), 
ing it. 

Approximately 1 to 1^ months appears to be needed by an instructor to plan a 
program and prepare printed instruction materials for each content focus, i.e., a 
command language or a database. Preparation time for two or more foci appears to 
be cumulative, for example, 2-4 months for a command language and a database. 

Complex databases require more time, for example, 3 months for a UDRI manual. 

Time needed to revise a program and its instruction materials can vary from 3 
days to 1 month dependent on the extent of the revision. By comparison, instruc-| 
tors can spend 10 to 100 hours to prepare one hour of computer-assisted instruction. 

Only the Pitt, ONTAP, and CRS programs provide each student with a terminal; 
in most other programs, two people share a terminal. Irrespective of the instrucj- 
tion methodology employed, all programs produce printed materials for students. 

These vary from very brief guides to a database and retrieval options (H.I.S.) to) 
replicas of entire lecture content (NLM). 

Six of the programs distribute questionnaires to students at the end of a 
program to obtain a course evaluation. No program has a longer term evaluation 
mechanism, for example, a follow-up six months after the course. The customer 
service desks maintained by LIRS, DTIC, NLM, and H.I.S. provide feedback to in¬ 
structors, but only for NLM is this direct since NLM instruction staff also per¬ 
form service desk duties. The user group meetings of DTIC and NLM and the update| 
session^ of LIRS are other opportunities for feedback. 

t t ^fj^ s “ on practice is a sine qua non for these instruction programs. Practice! 

° a n § 4 to 5 hours per student is suggested for programs having the objective 
of providing the student with proficiency to do online searches on-the-job. 

CRS's computer-based instruction (CBI) program demonstrates that the CBI 
methodology can be used to convey introductory-level knowledge of a corraaand lan¬ 
guage and databases. CBI is not yet hospitable to teaching advanced search com¬ 
plexities. It is also expensive. The LIRS ONTAP approach, that uses computers 
in a more limited way, is worthy of further study. It is self-instructional and,| 
with proper printed materials, it might be able to effectively replace up to 85% 
of present instruction programs. 
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